Papers with downstream tasks
DynaMaR: Dynamic Prompt with Mask Token Representation (2022.emnlp-industry)
Copied to clipboard
Xiaodi Sun, Sunny Rajagopalan, Priyanka Nigam, Weiyi Lu, Yi Xu, Iman Keivanloo, Belinda Zeng, Trishul Chilimbi
| Challenge: | Recent research shows that large language models pretrained using unsupervised approaches can achieve significant performance improvement on many downstream tasks. |
| Approach: | They propose an unsupervised approach to fine-tuning large language models using unsupervised approaches to many downstream tasks. |
| Outcome: | The proposed approach improves on four e-commerce applications and can achieve an average improvement of 10% in few-shot settings and 3.7% in data-rich settings over the standard approach. |
PoemBERT: A Dynamic Masking Content and Ratio Based Semantic Language Model For Chinese Poem Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Despite the lack of pre-trained models for ancient Chinese poetry, the unique artistry and structural nuances of Chinese poetry present complex challenges for machine learning in creative applications. |
| Approach: | They propose a BERT-based model incorporating sentiment and pinyin embeddings into the model, enhancing its sensitivity to emotional information and addressing challenges posed by the phenomenon of multiple pronunciations for the same Chinese character. |
| Outcome: | The proposed model outperforms existing models on poem generation and sentiment classification tasks and is state-of-the-art in automatic and manual evaluations. |
A Deeper Look into Dependency-Based Word Embeddings (N18-4)
Copied to clipboard
| Challenge: | Word embeddings trained with dependency contexts excel at different tasks, and enhanced dependencies often improve performance. |
| Approach: | They propose to use dependency-based word embeddings to capture semantic similarity rather than relatedness. |
| Outcome: | The results show that word embeddings trained with Universal and Stanford dependencies excel at different tasks and that enhanced dependencies often improve performance. |
GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work on Augmented Language Models (LLMs) over-rely on task-specific demonstrations that limits their generalizability and computational cost. |
| Approach: | They propose a query-tool grounding algorithm that is generalizable to various tasks . they delegate tool grounding and execution to small language models and LLMs . |
| Outcome: | The proposed algorithm outperforms baselines on 14 datasets and shows it can be generalized to different tasks. |
Vision-Language Pretraining: Current Trends and the Future (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Recent vision-language models are being used for downstream tasks that require large datasets and supervised datasets. |
| Approach: | They focus on recent vision-language pretraining paradigms and their strengths and shortcomings . they compare the different family of models used for vision- language pretraining . |
| Outcome: | This paper provides the background on image–language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. |
A Multilingual Reading Comprehension System for more than 100 Languages (2020.coling-demos)
Copied to clipboard
Anthony Ferritto, Sara Rosenthal, Mihaela Bornea, Kazi Hasan, Rishav Chakravarti, Salim Roukos, Radu Florian, Avi Sil
| Challenge: | Recent advances in open domain question answering (QA) have focused on machine reading comprehension (MRC) |
| Approach: | They propose a multilingual machine reading comprehension (MRC) demo which can answer questions in over 100 languages. |
| Outcome: | The proposed system can answer questions in over 100 languages and integrates with IBM Watson's machine translation widget to improve language accessibility. |
Improving and Simplifying Template-Based Named Entity Recognition (2023.eacl-srw)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is traditionally approached as a sequence labeling task where a tag is predicted for each token. |
| Approach: | They propose to convert a Named Entity Recognition task into a seq2seq task by generating synthetic sentences using templates. |
| Outcome: | The proposed model outperforms the current state-of-the-art approach in resource-rich, low resource and domain transfer settings and the negative examples play an important role in its performance. |
Evaluating Universal Dependency Parser Recovery of Predicate Argument Structure via CompChain Analysis (2021.starsem-1)
Copied to clipboard
| Challenge: | Compchains are a categorization of the hierarchy of predicate dependency relations present within a UD parse. |
| Approach: | They introduce compchains, a categorization of the hierarchy of predicate dependency relations present within a UD parse. |
| Outcome: | The proposed model performs poorly on sentences with predicate-argument structure with more than one level of embedding. |
Comparison and Combination of Sentence Embeddings Derived from Different Supervision Signals (2022.starsem-1)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings have not been well understood what properties are captured in the resulting sentences depending on the supervision signals. |
| Approach: | They propose to combine two types of sentence embedding methods with similar architectures and tasks to investigate their properties. |
| Outcome: | The proposed methods perform better on unsupervised and downstream tasks than the proposed methods on untrained STS tasks and probing tasks. |
Enhancing Adversarial Transferability in Visual-Language Pre-training Models via Local Shuffle and Sample-based Attack (2025.findings-naacl)
Copied to clipboard
| Challenge: | Visual-Language Pre-training (VLP) models are vulnerable to adversarial examples . previous studies have focused on improving adversariality of models . |
| Approach: | They propose a local shuffle and sample-based attack that randomly shufts one of the local image blocks and generates adversarial images and samples around them. |
| Outcome: | The proposed attack outperforms other advanced attacks on Large Vision-Language Models and outperformed previous attacks on Visual-Langue Pre-training models. |
ALAMBIC : Active Learning Automation Methods to Battle Inefficient Curation (2023.eacl-demo)
Copied to clipboard
| Challenge: | ALAMBIC is an open-source web-based platform for annotating text data through active learning for classification task. |
| Approach: | They present an open-source web-based platform for annotating text data through active learning for classification task. |
| Outcome: | The proposed model can be downloaded and used in downstream tasks and integrates with other types of models, features and active learning strategies. |
Multilingual BERT Post-Pretraining Alignment (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work improves on the success of monolingual pretrained language models by adding cross-lingual tasks that always involve English. |
| Approach: | They propose a method to align multilingual contextual embeddings as a post-pretraining step for improved cross-lingual transferability of pretrained language models. |
| Outcome: | The proposed model outperforms XLM-R_Base on translation-train tasks while using less parallel data and fewer parameters. |
Chinese Grammatical Correction Using BERT-based Pre-trained Model (2020.aacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-trained models improve performance on downstream tasks. |
| Approach: | They propose to incorporate a pre-trained model into an encoder-decoder model to improve the performance of Chinese grammatical error correction tasks. |
| Outcome: | The proposed method improves the performance of Chinese grammatical error correction tasks. |
Knowledge extraction from aeronautical messages (NOTAMs) with self-supervised language models for aircraft pilots (2022.naacl-industry)
Copied to clipboard
| Challenge: | During pre-flight briefings, aircraft pilots analyse a long list of NOTAMs . the messages are usually written in the English language, but the phrasing is very special . |
| Approach: | They pretrain language models derived from BERT on circa 1 million unlabeled NOTAMs . they reuse the learnt representations on three downstream tasks valuable for pilots - criticality prediction, named entity recognition and translation into a structured language called Airlang. |
| Outcome: | The proposed language model can be used on criticality prediction, named entity recognition and translation into a structured language called Airlang. |
SwahBERT: Language Model of Swahili (2022.naacl-main)
Copied to clipboard
| Challenge: | Social media and Internet forums are valuable sources of citizens’ opinions, which can be analyzed for community development and user behavior analysis. |
| Approach: | They present a pre-training and annotated datasets of Swahili and an emotion classification datasets that are manually annotating by two native Swahils. |
| Outcome: | The proposed model outperforms existing monolingual language model in almost all downstream tasks. |
Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages (2025.acl-srw)
Copied to clipboard
| Challenge: | Low-resource languages (LRLs) face significant challenges in natural language processing due to limited data. |
| Approach: | They evaluate adapter-based methods for adapting mLMs to low-resource languages . they use unstructured text and structured knowledge from ConceptNet to evaluate adapters . |
| Outcome: | The proposed methods outperform large language models and LLaMA-3 and deepSeek-R1 models on low training data. |
Mind Your Special Tokens! On the Importance of Dedicated Sequence-End Tokens in Vision-Language Embedding Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) are highly sensitive to end-of-input artifacts in fine-tuning and inference data, e.g., whether input sequences end with punctuation or newline characters. |
| Approach: | They propose to convert generative LVLMs into vision-language encoders via contrastive learning objectives and use supervised contrastive objectives to train them. |
| Outcome: | The proposed approach improves visual and text representations and improves retrieval and (semantic) similarity tasks. |
LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing benchmarks for natural language processing focus on understanding or generating short texts . lack of standardized benchmarks makes it difficult to assess and compare models . |
| Approach: | They propose a story-centric benchmark for Chinese long text modeling that aggregates two understanding tasks and two generation tasks. |
| Outcome: | The proposed model outperforms similar-sized models on understanding and generation tasks. |
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2022.acl-long)
Copied to clipboard
| Challenge: | Existing pretraining frameworks do not perform well for all tasks of three main categories, such as natural language understanding (NLU), unconditional generation, and conditional generation. |
| Approach: | They propose a general language model based on autoregressive blank infilling to address this challenge. |
| Outcome: | The proposed model outperforms BERT, T5, and GPT on a wide range of tasks across NLU, conditional and unconditional generation tasks. |
OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models (2023.acl-demo)
Copied to clipboard
| Challenge: | Existing implementations that modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM have limited the practicality and flexibility of delta tuning. |
| Approach: | They propose an open-source library that provides a plug-and-play implementation of delta tuning methods for pre-trained models. |
| Outcome: | The proposed methods eliminate the need to modify the backbone PTMs’ code, making OpenDelta compatible with different, even novel PTM. |
Relation Embedding with Dihedral Group in Knowledge Graph (P19-1)
Copied to clipboard
| Challenge: | Existing models for link prediction fail to capture relation compositions, resulting in lack of interpretability for reasoning on incomplete knowledge graph (KG). |
| Approach: | They propose a new model that learns knowledge graph embeddings that can capture relation compositions by nature and reduces the solution space drastically. |
| Outcome: | The proposed model outperforms existing models and is comparable to or better than deep learning models such as ConvE. |
Revealing Single Frame Bias for Video-and-Language Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for video-and-language learning use multiple frames as inputs. |
| Approach: | They propose to use single-frame models for video-and-language learning to investigate temporality in video- and language tasks. |
| Outcome: | The proposed model does not take into account temporal information on video-and-language tasks. |
HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models (2022.findings-aacl)
Copied to clipboard
| Challenge: | Existing methods do not examine social groups categorised by geographical information, leaving the region-related biases in pre-trained LMs unexplored. |
| Approach: | They propose a hierarchical regional bias evaluation method to quantify regional bias in pre-trained language models. |
| Outcome: | The proposed method evaluates regional bias with regard to comprehensive topics and measures potential regional bias that can be propagated to downstream tasks. |
STEP: Staged Parameter-Efficient Pre-training for Large Language Models (2025.naacl-short)
Copied to clipboard
| Challenge: | Recent LLM development trends involve pre-training models with a vast number of parameters on massive datasets. |
| Approach: | They propose a method that integrates parameter-efficient tuning techniques with model growth to reduce memory requirements while maintaining equivalent performance. |
| Outcome: | The proposed method reduces memory requirements by 53.9% while maintaining equivalent performance to vanilla pre-trained models on downstream tasks. |
Diachronic degradation of language models: Insights from social media (P18-2)
Copied to clipboard
| Challenge: | Existing studies have explored whether and how language models degrade over time, i.e. why they fail to work on contemporary language. |
| Approach: | They investigate the accuracy of pre-trained language models for downstream tasks in machine learning and user profiling. |
| Outcome: | The results show that it is possible to measure diachronic drifts within social media and within the span of a few years. |
Meta-learning Pathologies from Radiology Reports using Variance Aware Prototypical Networks (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods for fewshot learning require a large number of in-domain labeled examples for fine tuning. |
| Approach: | They propose to extend the Prototypical Networks for few-shot text classification by replacing Gaussian class prototypes with a regularization term that encourages the examples to be clustered near the appropriate class centroids. |
| Outcome: | The proposed method outperforms baselines on 13 public and 4 internal datasets and detects potential out-of-distribution (OOD) data points during deployment. |
NepBERTa: Nepali Language Model Trained in a Large Corpus (2022.aacl-short)
Copied to clipboard
| Challenge: | Nepali is a low-resource language with more than 40 million speakers worldwide. |
| Approach: | They present a BERT-based natural language understanding model trained on the most extensive monolingual Nepali corpus ever. |
| Outcome: | The proposed model performs well in Nepali-specific NLP tasks including Named-Entity Recognition, Content Classification, POS Tagging, and Sequence Pair Similarity. |
Pruning Pre-trained Language Models Without Fine-Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to prune Pre-trained Language Models (PLMs) are overparameterized and require fine-tuning. |
| Approach: | They propose a pruning method that uses first-order pruning to prune PLMs while fine-tuning the remaining weights. |
| Outcome: | The proposed method outperforms first-order pruning and zero-order methods at sparsity levels. |
CaRe: Open Knowledge Graph Embeddings (D19-1)
Copied to clipboard
| Challenge: | Existing methods for generating Open Knowledge Graphs have been criticized for not achieving canonicalization of OpenKGs. |
| Approach: | They propose to use Open Information Extraction methods to extract triples from text . they propose to learn embeddings of NPs and RPs present in the graph . |
| Outcome: | The proposed methods improve OpenKG embeddings and bootstrap OpenKGs from text corpus. |
Towards Unified Prompt Tuning for Few-shot Text Classification (2022.findings-emnlp)
Copied to clipboard
Jianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan, Minghui Qiu, Fei Yang, Qiuhui Shi, Songfang Huang, Ming Gao
| Challenge: | Prompt-based fine-tuning has boosted performance of Pre-trained Language Models (PLMs) on few-shot text classification, but PLMs are unfamiliar with prompt-style expressions during pre-training, which limits the few- shot learning performance on downstream tasks. |
| Approach: | They propose a framework for prompt-based fine-tuning that captures prompting semantics from non-target NLP datasets and propose 'Prompt-Options-Verbalizer' for joint prompt learning across different NLP tasks. |
| Outcome: | Experiments show that the proposed framework outperforms state-of-the-art prompt-based fine-tuning frameworks on few-shot text classification tasks. |
FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Lowrank adaptation and its variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. |
| Approach: | They propose a framework to fine-tune LoRA modules on a large-scale instruction tuning dataset. |
| Outcome: | The proposed framework outperforms existing PEFT methods and significantly reduces inference latency. |
PRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation (2021.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to building task-oriented dialog systems require a substantial amount of annotations and thus are labor-intensive. |
| Approach: | They propose a Pre-trainedRole Alternating Language model (PRAL) that is explicitly designed for task-oriented dialog tasks. |
| Outcome: | The proposed model outperforms or is on par with state-of-the-art models on task-oriented dialog tasks. |
C-MORE: Pretraining to Answer Open-Domain Questions by Consulting Millions of References (2022.acl-short)
Copied to clipboard
| Challenge: | Existing approaches to pretrain open-domain question answering systems lack task-specific annotations. |
| Approach: | They propose to pretrain a two-stage open-domain question answering system with strong transfer capabilities by using a dictionary and a large-scale corpus. |
| Outcome: | The proposed approach leads to 2%-10% gains in top-20 accuracy and improves with reader. |
A Fully Hyperbolic Neural Model for Hierarchical Multi-Class Classification (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for fine-grained entity typing have a hierarchical structure . prior work has integrated only explicit hierarchic information by formulating a hierarchy-aware loss or by representing instances and labels in a joint Euclidean embedding space. |
| Approach: | They propose a fully hyperbolic model for multi-class multi-label classification that performs all operations in hyperbolical space. |
| Outcome: | The proposed model performs all operations in hyperbolic space on two challenging datasets and shows it is comparable to state-of-the-art methods on fine-grained classification with remarkable reduction of parameter size. |
Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing probing datasets focus on knowledge about relations between words and entities. |
| Approach: | They propose a new probing task to evaluate word understanding directly using dictionary definitions of words. |
| Outcome: | The proposed task evaluates word understanding directly using dictionary definitions of words. |
Building Location Embeddings from Physical Trajectories and Textual Representations (2020.aacl-main)
Copied to clipboard
| Challenge: | Using a dataset consisting of the location trajectories of 729 students over a seven month period, we investigate whether embeddings can represent aspects such as location presence or location functionality. |
| Approach: | They propose to use location embeddings to generate embeddables of sequences of locations a student has visited to identify surface properties captured in the representations. |
| Outcome: | The proposed models can be used to predict depression levels and area of study, and can be applied to complex tasks such as predicting area of studies and depression levels. |
Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large multi-modal models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across multi-dimensional applications. |
| Approach: | They propose a parameter-efficient fine-tuning strategy that combines both . they find that parameter tuning methods distort the feature representation space . |
| Outcome: | The proposed strategy preserves representation space while limiting performance on downstream tasks. |
The Lifecycle of “Facts”: A Survey of Social Bias in Knowledge Graphs (2022.aacl-main)
Copied to clipboard
| Challenge: | Knowledge graphs are used in a variety of downstream tasks and in hybrid AI systems. |
| Approach: | They propose to examine the lifecycle of knowledge graphs with respect to bias influences. |
| Outcome: | The proposed models are based on the lifecycle of knowledge graphs and their embedded versions . they show that the KGs manifest biases and propagate harmful prejudices . |
Enhancing Tabular Reasoning with Pattern Exploiting Training (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing methods based on pre-trained language models have shown superior performance over tabular tasks despite showing inherent problems such as not using the right evidence and inconsistent predictions across inputs. |
| Approach: | They utilize Pattern-Exploiting Training (PET) on pre-trained language models to strengthen tabular reasoning models’ pre-existing knowledge and reasoning abilities. |
| Outcome: | The proposed model exhibits superior understanding of knowledge facts and tabular reasoning compared to baseline models. |
LM-CORE: Language Models with Contextually Relevant External Knowledge (2022.findings-naacl)
Copied to clipboard
| Challenge: | Large pre-trained language models can capture factual knowledge in their parameters but storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of information and resource requirements. |
| Approach: | They propose a framework that provides explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art knowledge-enhanced language models on knowledge probing tasks and can handle knowledge updates. |
Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning (2024.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that fine-tuning of large language models for specific tasks can be challenging . distribution shift during fine-timing can lead to performance degradation in general task capabilities . |
| Approach: | They propose a new approach that bridges the distribution gap between task datasets and LLMs by guiding fine-tuning with a distilled dataset generated by the model itself. |
| Outcome: | The proposed approach achieves comparable or superior performance on downstream tasks compared to the vanilla approach. |
Disentangling Representations of Text by Masking Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained models such as BERT encode a range of features into monolithic vectors, providing strong predictive accuracy across downstream tasks. |
| Approach: | They explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects. |
| Outcome: | The proposed method disentangles sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics. |
Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to perform named entity recognition (NER) on unlabeled data are difficult to obtain in low-resource domains. |
| Approach: | They propose ways to use unlabeled data for pretraining to improve performance in downstream tasks. |
| Outcome: | The proposed methods outperform models trained on unlabeled data on seven domains. |
Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing approaches to fine-tune pre-trained language models for downstream tasks require labeled data. |
| Approach: | They propose to self-train pre-trained language models to improve performance on data-scarce varieties by as large as 10% F1 and 2% accuracy. |
| Outcome: | The proposed model improves zero-shot MSA-to-DA transfer by as large as 10% F1 (NER) and 2% accuracy (POS tagging). |
Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization (2024.emnlp-main)
Copied to clipboard
| Challenge: | minimizing reconstruction error is not always ideal and can overfit calibration data. |
| Approach: | They propose a method to prune large language models by divide and conquer . they propose minimizing reconstruction error by more than 90% by using calibration data . |
| Outcome: | The proposed pruning approach generates high reconstruction errors . the proposed technique reduces reconstruction error by more than 90% . |
XDoc: Unified Pre-training for Cross-Format Document Understanding (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pre-trained models target one document format at a time, making it difficult to combine knowledge from multiple document formats. |
| Approach: | They propose a unified pre-trained model which deals with different document formats in a single model. |
| Outcome: | The proposed model achieves comparable or even better performance on a variety of downstream tasks compared with the individual pre-trained models. |
Protein-STORY: Semantic Text-Oriented Representation Yields biologically meaningful Protein embeddings (2026.acl-short)
Copied to clipboard
| Challenge: | Unsupervised representation learning relying on sequence data often overlooks decades of expert-curated biological knowledge stored in textual formats. |
| Approach: | They propose a pipeline that synthesizes protein embeddings from diverse, multi-source text descriptions and a network architecture that integrates high-fidelity functional and structural insights into a unified representation. |
| Outcome: | The proposed pipeline outperforms existing models on diverse downstream tasks (+2 pts F1) and enables zero-shot text-prompted protein search. |
Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for prompt tuning can overfit to few-shot training samples, causing overfitting . authors propose a new framework for prompt learning with supervised meta-learning . |
| Approach: | They propose a self-supervised meta-prompt learning framework with MEta-gradient Regularization for few-shot generalization that leverages self-recognized meta-learning with a diverse set of meta-tasks to learn a universal prompt initialization using only unlabeled data. |
| Outcome: | The proposed framework learns a universal prompt initialization for efficient adaptation using only unlabeled data. |
Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks (2024.findings-eacl)
Copied to clipboard
| Challenge: | Prior work on multimodal content classification has not addressed these challenges. |
| Approach: | They propose to use two auxiliary tasks to fine-tune multimodal models to address hidden cross-modal semantics and weak image-text relationships when modeling text and images. |
| Outcome: | The proposed model improves by up to 2.6 F1 score across five diverse social media datasets. |
Automatic Evaluation of Local Topic Quality (P19-1)
Copied to clipboard
Jeffrey Lund, Piper Armstrong, Wilson Fearn, Stephen Cowley, Courtni Byun, Jordan Boyd-Graber, Kevin Seppi
| Challenge: | Topic models are evaluated with global topic distributions but without local topic assignments. |
| Approach: | They propose a task to elicit human judgments of token-level topic assignments . they propose to use global metrics to evaluate topic models at a local level . |
| Outcome: | The proposed task elicits human judgments of token-level topic assignments . global metrics agree poorly with human assignments, the authors show . |
CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on code representation learning, which aims to represent the semantics of source code into distributed vectors. |
| Approach: | They propose to integrate different views with the natural-language description of source code into a unified framework with Multi-View contrastive Pre-training. |
| Outcome: | The proposed model outperforms state-of-the-art models on three downstream tasks over five datasets. |
Feature-Adaptive and Data-Scalable In-Context Learning (2024.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a popular way to stimulate LLM capabilities for downstream tasks due to context length constraints. |
| Approach: | They propose a feature-adaptive and data-scalable in-context learning framework which leverages task-adaptives to promote inference on the downstream task. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on 10 datasets under different data settings and LLM scale. |
Judging the Judges: Can Large Vision-Language Models Fairly Evaluate Chart Comprehension and Reasoning? (2025.acl-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Ahmed Masry, Mizanur Rahman, Amran Bhuiyan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) are expensive and time-consuming to evaluate . however, they are limited in their use in industrial settings due to their limited availability and limited resources. |
| Approach: | They evaluate 13 open-source LVLMs as judges for diverse chart comprehension and reasoning tasks. |
| Outcome: | The proposed models can be used to assess chart comprehension and reasoning tasks, but they are expensive and time-consuming. |
Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual BERT (mBERT) has shown reasonable capability for zero-shot cross-lingual transfer when fine-tuned on downstream tasks. |
| Approach: | They propose to use parallel corpora and rotational alignment methods to improve transfer performance in a zero-shot setting. |
| Outcome: | The proposed method improves rotation-based alignment on Name Entity Recognition and Semantic Slot Filling tasks. |
VividMed: Vision Language Model with Versatile Visual Grounding for Medicine (2025.naacl-long)
Copied to clipboard
| Challenge: | Vision Language Models (VLMs) have demonstrated promise in generating visually grounded responses, but their application in the medical domain is hindered by unique challenges. |
| Approach: | They propose a vision language model with versatile visual grounding for medicine that generates semantic segmentation masks and instance-level bounding boxes. |
| Outcome: | The proposed model can generate semantic segmentation masks and instance-level bounding boxes, and accommodates various imaging modalities, including both 2D and 3D data. |
Multi-Task Retrieval for Knowledge-Intensive Tasks (2021.acl-long)
Copied to clipboard
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, Gargi Ghosh
| Challenge: | Knowledge-intensive tasks require large amounts of knowledge about the world . recent neural retrieval models achieve better results by learning directly from task-specific training data. |
| Approach: | They propose a multi-task trained neural retrieval model that can be universally trained on a wide variety of problems. |
| Outcome: | The proposed model outperforms specialised retrievers on a few-shot setting and matches or improves state-of-the-art on multiple benchmarks. |
ITERATE: Image-Text Enhancement, Retrieval, and Alignment for Transmodal Evolution with LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | a new framework for visual annotation of text-based questions is needed to improve performance . obtaining corresponding images through manual annotation often entails high costs . |
| Approach: | They propose a framework that uses visual modality to enhance the performance of text-based questions. |
| Outcome: | The proposed framework improves the alignment between text and images by using search engines or web scraping techniques. |
Event Semantic Classification in Context (2024.findings-eacl)
Copied to clipboard
| Challenge: | In this work, we focus on the semantic classification of events in context to help machines gain a deeper understanding of events. |
| Approach: | They propose to integrate event semantics into downstream tasks to help machines understand events better. |
| Outcome: | The proposed model improves the understanding of events in context. |
Zero-Shot Information Extraction as a Unified Text-to-Triple Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | a number of information extraction tasks require task-specific training. |
| Approach: | They propose a text-to-triple translation framework for information extraction tasks . they propose enabling task-agnostic translation by leveraging latent knowledge of a pre-trained language model . |
| Outcome: | The proposed framework outperforms the existing methods on open information extraction tasks. |
Composite Backdoor Attacks Against Large Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated superior performance on various tasks, but untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks. |
| Approach: | They propose a composite backdoor attack that scatters multiple trigger keys in different prompt components. |
| Outcome: | The proposed attack achieves 100% Attack Success Rate (ASR) with a False Triggered Rate (FTR) below 2.06% and negligible model accuracy degradation. |
Language Model Pre-Training with Sparse Latent Typing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Modern large-scale Pre-trained Language Models focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences. |
| Approach: | They propose a new pre-training objective that enables the model to learn latent types . the objective allows the model a self-supervised way to extract sentence-level keywords . |
| Outcome: | The proposed model learns interpretable latent type categories without external knowledge and improves downstream tasks. |
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models use the attention mechanism to contextualize input inputs . but, we find that it is not as important as thought for pretrained models . |
| Approach: | They propose a probing method that replaces input-dependent attention matrices with constant ones. |
| Outcome: | The proposed method improves performance of pretrained language models without input-dependent attention. |
AoE: Angle-optimized Embeddings for Semantic Textual Similarity (2024.acl-long)
Copied to clipboard
| Challenge: | Existing text embedding models neglect cosine saturation zones rendering vanishing gradients and hindering learning subtle semantic differences in text embeds. |
| Approach: | They propose a novel angle-optimized Embedding model that optimizes angle differences in complex space to explore similarity in saturation zones better. |
| Outcome: | The proposed model outperforms existing models neglecting cosine saturation zones on STS and MTEB benchmarks and can produce high-quality embeddings and benefit downstream tasks. |
Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization (2021.naacl-main)
Copied to clipboard
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula
| Challenge: | Large-scale, open Natural Language Inference datasets have catalyzed the development of NLI models that exhibit close to human-level performance, but the use of these models for other downstream NLP tasks has met with limited success. |
| Approach: | They use multiple-choice reading comprehension and checking factual correctness of textual summarization tasks to investigate potential reasons for this . authors leverage abundance of data from reading comprehension datasets into longer-premise NLI datasets to improve their models . |
| Outcome: | The proposed models outperform models trained on converted datasets due to the difference in premise lengths. |
LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin (2024.acl-long)
Copied to clipboard
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Experimental results show that, as the instruction data increases, LoRAMoE can significantly improve the ability to process downstream tasks, while maintaining the world knowledge stored in the LLM. |
| Approach: | They propose a framework that introduces several low-rank adapters and integrates them by using a router network to freeze the backbone model and force a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks. |
| Outcome: | The proposed framework freezes the backbone model and forces a portion of LoRAs to focus on leveraging world knowledge to solve downstream tasks, to alleviate world knowledge forgetting. |
End-to-End Graph-Based TAG Parsing with Neural Networks (N18-1)
Copied to clipboard
| Challenge: | Using BiLSTMs, highway connections, and character-level CNNs, we propose a graph-based Tree Adjoining Grammar (TAG) parser. |
| Approach: | They propose a graph-based Tree Adjoining Grammar parser that uses BiLSTMs, highway connections, and character-level CNNs. |
| Outcome: | The proposed parser outperforms the previously reported best by more than 2.2 LAS and UAS points. |
Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-Efficient Fine-Tuning (PEFT) methods have gained popularity for adapting pre-trained Large Language Models (LLMs) to downstream tasks. |
| Approach: | They propose a method to optimize the importance of full layers with layer-wise importance scoring by leveraging the estimated importance scores. |
| Outcome: | The proposed method is compatible with PEFT methods that operate on a per-layer basis and achieves better performance. |
Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text (2025.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling is widely adopted, but the process of selecting tokens for masking is random and the percentage of masked tokens is typically fixed for the entire training process. |
| Approach: | They propose to adjust the masking ratio based on a task-informed anti-curriculum learning scheme to mask useful and harmful tokens. |
| Outcome: | The proposed approach improves the ability of the model to focus on key task-relevant features, contributing to statistically significant performance gains across tasks. |
Exploring Data Augmentation for Code Generation Tasks (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have impacted how models are trained for programming language tasks. |
| Approach: | They propose to use augmentation methods that yield consistent improvements in code translation and summarization by up to 6.9% and 7.5% respectively. |
| Outcome: | The proposed methods improve translation and summarization by 6.9% and 7.5% respectively. |
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent models for learning discourse language representations focus on bottom or top-level representations, but they do not capture intermediate-size structures in natural languages such as sentences and the relationships among them. |
| Approach: | They propose a new objective for learning a discourse language representation in a self-supervised manner by shuffling the sequence of input sentences and training a hierarchical transformer model to reconstruct the original ordering. |
| Outcome: | The proposed model improves the original BERT model on downstream tasks by large margins. |
Pruning Adatperfusion with Lottery Ticket Hypothesis (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive to fine-tune and require large storage. |
| Approach: | They propose a method to identify the influence of each adapter module and a way to prune adapters based on the Lottery Ticket Hypothesis. |
| Outcome: | The proposed model reduces size significantly while keeping performance intact. |
Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent (2025.findings-emnlp)
Copied to clipboard
Junda Wu, Yuxin Xiong, Xintong Li, Yu Xia, Ruoyu Wang, Yu Wang, Tong Yu, Sungchul Kim, Ryan A. Rossi, Lina Yao, Jingbo Shang, Julian McAuley
| Challenge: | Existing fine-tuning and continual learning methods compress visual representations and emphasize task alignment over visual retention. |
| Approach: | They propose a modality-decoupled gradient descent (MDGD) that regulates gradient updates to preserve effective rank of visual features and explicitly disentangles visual learning from task-specific alignment. |
| Outcome: | The proposed model reduces visual forgetting and improves visual retention . it disentangles visual learning from task-specific alignment and preserves effective rank . |
Jigsaw Pieces of Meaning: Modeling Discourse Coherence with Informed Negative Sample Synthesis (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing work on creating “informed” incoherent samples for coherence modeling has focused on permutations of a coherent document . |
| Approach: | They propose to use Constituency trees, Part-of-speech, semantic overlap to create “informed” negative samples that better represent or mimic incoherence. |
| Outcome: | The proposed methods improve the quality of the negative sample. |
Robustness Challenges in Model Distillation and Pruning for Natural Language Understanding (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on compressing pre-trained language models (PLMs) however, few studies have examined the impact of compression on generalizability and robustness of compressed models for out-of-distribution data. |
| Approach: | They propose to use knowledge distillation and pruning to reduce model generalization and generalization on out-of-distribution data. |
| Outcome: | The proposed compression techniques overfit on shortcut samples and generalize poorly on hard ones. |
GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that Pre-trained Language Models store relational knowledge and utilize it for performing downstream tasks. |
| Approach: | They propose a benchmark dataset for probing the diversity of relational knowledge in multilingual PLMs. |
| Outcome: | The proposed dataset contains 3125 prompts in English, Chinese, Hindi, Persian, and Swahili . larger multilingual PLMs variants do not store geo-diverse concepts better than its smaller variant . |
K-Merge: Online Continual Merging of Adapters for On-device Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful general-purpose models that can be adapted to a wide range of problem types in many languages. |
| Approach: | They propose a method for on-device online continual merging to integrate new LoRAs when a new one becomes available. |
| Outcome: | The proposed approach outperforms other methods while adhering to storage budget constraints. |
Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing financial PLMs are not pretrained on sufficiently diverse financial data, leading to subpar generalization performance. |
| Approach: | They propose to pretrain financial PLMs on financial corpus and train financial models on financial data. |
| Outcome: | The proposed financial language models outperform existing financial PLMs on financial tasks even for unseen corpus groups. |
Contextual String Embeddings for Sequence Labeling (C18-1)
Copied to clipboard
| Challenge: | Recent advances in language modeling have made it viable to model language as distributions over characters. |
| Approach: | They propose to leverage internal states of a trained character language model to produce a new type of word embeddings. |
| Outcome: | The proposed embeddings outperform the state-of-the-art on four classic sequence labeling tasks. |
Prompt Candidates, then Distill: A Teacher-Student Framework for LLM-driven Data Annotation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for data annotation use an aggressive approach prompting LLMs to determine a single gold label for each unlabeled sample. |
| Approach: | They propose a teacher-student framework that distills candidate annotations with a Small Language Model (SLM) they propose to use LLMs to generate and distill candidate annotation with slms to ensure unique labels are provided for downstream tasks. |
| Outcome: | The proposed method outperforms existing methods due to uncertainty in LLMs and is noisetolerant. |
Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to assigning ICD codes to clinical text are time-consuming, labor intensive, and error-prone. |
| Approach: | They propose to adapt a Transformer-based model to a longformer model and use it to encode clinical narratives. |
| Outcome: | The proposed approach outperforms current state-of-the-art models in ICD coding with the label embeddings contributing to the good performance. |
Collaborative Performance Prediction for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are one of the most important AI research powered by largescale parameters, high computational resources, and massive training data. |
| Approach: | They propose a framework that leverages historical performance of large language models and other design factors to improve prediction accuracy. |
| Outcome: | The proposed framework surpasses scaling laws in predicting performance of large language models . it also facilitates a detailed analysis of factor importance, an area previously overlooked . |
Intrinsic Bias Metrics Do Not Correlate with Application Bias (2021.acl-long)
Copied to clipboard
| Challenge: | a recent survey of bias in natural language processing found that a coreference system makes more errors in an anti-stereotypical coreferent than in a pro-sterereotype one. |
| Approach: | They compare intrinsic and extrinsic bias metrics across hundreds of trained models . they urge researchers to focus on extrindic measures of bias, not easy to measure . |
| Outcome: | a new intrinsic metric and an annotated test set on gender bias in hate speech are tested . authors urge researchers to focus on extrinsic measures of bias, and to make them more feasible . |
Towards Lossless Encoding of Sentences (P19-1)
Copied to clipboard
| Challenge: | Existing methods for encoding text into lossless representations focus on performing well on downstream tasks and are unable to reconstruct original sequence from learned embedding. |
| Approach: | They propose a lossless method for encoding long sequences of texts into feature rich representations by recursive autoencoding. |
| Outcome: | The proposed method performs well on sentiment analysis and sentiment classification tasks. |
Revisiting Automated Prompting: Are We Actually Doing Better? (2023.acl-short)
Copied to clipboard
| Challenge: | Recent work demonstrates that Large Language Models are great few-shot learners, and prompting significantly increases their performance on a range of downstream tasks. |
| Approach: | They revisit techniques for automated prompting on six different downstream tasks and a larger range of K-shot learning settings. |
| Outcome: | The proposed approach outperforms manual prompting on six different downstream tasks and a larger range of K-shot learning settings. |
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models perform well on commonsense-reasoning benchmark datasets, but building machines with commonsence to compose plausible sentences remains challenging. |
| Approach: | They propose a constrained text generation task for generative commonsense reasoning that generates a coherent sentence using common concepts. |
| Outcome: | The proposed task generates a coherent sentence describing an everyday scenario using common concepts over 35k concept-sets. |
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets (2021.acl-long)
Copied to clipboard
| Challenge: | Large-scale pre-trained language models require enormous computational resources and long training time. |
| Approach: | They propose an algorithm to reduce inference time and train large NLP models by slimming the self-attention and fully-connected sub-layers inside a transformer. |
| Outcome: | The proposed algorithm achieves comparable performance to standard BERT with 35 45% less training time. |
Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on dialogue modeling use pre-trained language models to encode dialogue history as successive tokens, which is insufficient in capturing the temporal characteristics of dialogues. |
| Approach: | They propose a bidirectional information decoupling network as a universal dialogue encoder which explicitly incorporates both the past and future contexts. |
| Outcome: | The proposed model incorporates past and future contexts and can be generalized to a wide range of dialogue-related tasks. |
BiKT: Enabling Bidirectional Knowledge Transfer Between Pretrained Models and Sequential Downstream Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks adapt from initial pretrained model to each downstream task directly, but ignore sequential nature of downstream tasks and feedback effect on pretrained models. |
| Approach: | They propose a framework to enable bidirectional knowledge transfer between pretrained models and downstream tasks in rounds. |
| Outcome: | The proposed framework improves on 9 GLUE datasets and 6 SuperGLUEs. |
Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing dynamic topic models lack the ability to reveal the evolution of topics . Existing models suffer from repetitive topic and unassociated topic issues . |
| Approach: | They propose a new evolution-tracking contrastive learning method that builds the similarity relations among dynamic topics and an unassociated word exclusion method to avoid unassociated topics. |
| Outcome: | The proposed model outperforms state-of-the-art models on downstream tasks and is robust to evolution intensities. |
ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing quality filtering methods rely on a high-quality dataset as reference . Existing methods introduce potential biases and compromise diversity . |
| Approach: | They propose a method that evaluates text quality based on the perplexity difference between two language models trained on the same data. |
| Outcome: | The proposed approach improves performance of pre-trained models without increasing training costs. |
Timeline-based Sentence Decomposition with In Context Learning for Temporal Fact Extraction (2024.acl-long)
Copied to clipboard
| Challenge: | Recent research on temporal fact extraction fails to establish time-to-fact correspondences in complex sentences. |
| Approach: | They propose a timeline-based sentence decomposition strategy using large language models with in-context learning to extract temporal facts from natural language text. |
| Outcome: | The proposed method achieves state-of-the-art on a complex temporal fact extraction dataset. |
Bridging Argument Quality and Deliberative Quality Annotations with Adapters (2023.findings-eacl)
Copied to clipboard
| Challenge: | Assessing the quality of an argument is a complex, highly subjective task . argument quality dimensions are complex and dependent on the context in which it is assessed . |
| Approach: | They propose a multi-task learning framework that incorporates knowledge about related dimensions into the learning process. |
| Outcome: | The proposed framework improves quality prediction in an extrinsic, out-of-domain task. |
Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach (2025.coling-main)
Copied to clipboard
| Challenge: | Using language models (LMs) has increased in use, and the use of biases and stereotypes is creating social problems. |
| Approach: | They propose a method to mitigate LM biases by continual training on biased data . they use masked language modeling to construct a Bias Vector as the difference between biased LMs and pre-trained LM weights . |
| Outcome: | The proposed method improves on the GLUE and SEAT benchmarks. |
Syntax-guided Contrastive Learning for Pre-trained Language Model (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies rely on additional syntax-driven attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. |
| Approach: | They propose a syntax-guided contrastive learning method which does not change the transformer architecture and does not alter the transformer structure. |
| Outcome: | The proposed method achieves consistent improvements in a variety of tasks including grammatical error detection, entity tasks, structural probing and GLUE. |
Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning (2024.findings-emnlp)
Copied to clipboard
Mohammed Saidul Islam, Raian Rahman, Ahmed Masry, Md Tahmid Rahman Laskar, Mir Tafseer Nayeem, Enamul Hoque
| Challenge: | Recent studies have demonstrated that large vision language models (LVLMs) are not multi-modal and lack multi-tasking capabilities. |
| Approach: | They evaluate the performance of large vision language models (LVLMs) for chart understanding and reasoning tasks and compare them to open-source models. |
| Outcome: | The proposed models demonstrate impressive abilities in generating fluent texts covering high-level data insights, but they also encounter common problems like hallucinations, factual errors, and data bias. |
Partially-Random Initialization: A Smoking Gun for Binarization Hypothesis of BERT (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained BERT has been used for natural language processing tasks but its performance is limited by memory and computational complexity. |
| Approach: | They propose to use pre-trained BERT to achieve decent accuracy . they propose to combine binary BERT with a randomly-initialized encoder . |
| Outcome: | The proposed model achieves state-of-the-art on GLUE and SQuAD benchmarks. |
A Multi-Task Embedder For Retrieval Augmented LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing retrieval methods face limitations in terms of knowledge, memory, and action. |
| Approach: | They propose a retrieval enhancement mechanism that brings in useful information from external sources to augment the LLM. |
| Outcome: | The proposed method significantly improves the LLM’s performance in various downstream tasks while introducing superior retrieval augmentation’s effect over both general and task-specifc retrievers. |
Unity in Diversity: Collaborative Pre-training Across Multimodal Medical Sources (2024.acl-long)
Copied to clipboard
Xiaochen Wang, Junyu Luo, Jiaqi Wang, Yuan Zhong, Xiaokun Zhang, Yaqing Wang, Parminder Bhatia, Cao Xiao, Fenglong Ma
| Challenge: | Current pre-training techniques rely on a limited scope of medical data, limiting the range of downstream tasks. |
| Approach: | They propose a pre-training strategy that unifies patient data within individual sources and captures explicit and implicit correlations between patients across different sources. |
| Outcome: | The proposed strategy bridges the gap between multimodal medical sources by aggregating patient data within individual sources and capturing explicit and implicit correlations between patients across sources. |
Conditional Supervised Contrastive Learning for Fair Text Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have demonstrated societal bias in existing NLP models. |
| Approach: | They propose to use contrastive learning to learn fair representations for text classification . they conduct experiments on two text datasets to demonstrate their methods are stable . |
| Outcome: | The proposed methods balancing task performance and bias mitigation are stable in different hyperparameter settings. |
Parameter-efficient Modularised Bias Mitigation via AdapterFusion (2023.eacl-main)
Copied to clipboard
Deepak Kumar, Oleg Lesota, George Zerveas, Daniel Cohen, Carsten Eickhoff, Markus Schedl, Navid Rekabsaz
| Challenge: | Large pre-trained language models contain societal biases and carry along these biase . Current approaches to mitigate these bias impose debiasing by updating model parameters, effectively transferring model to irreversible debiased state. |
| Approach: | They propose to develop stand-alone debiasing functionalities separate from the model, which can be integrated into the model on-demand while keeping the core model untouched. |
| Outcome: | The proposed approach improves or maintains effectiveness of bias mitigation, avoids catastrophic forgetting in a multi-attribute scenario, and maintains on-par task performance while granting parameter-efficiency and easy switching between the original and debiased models. |
UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning (2021.acl-long)
Copied to clipboard
| Challenge: | Existing pre-training methods focus on single-modal tasks or multi-modal ones . large-scale pre- training has drawn much attention in both the community of Compute Vision (CV) and Natural Language Processing (NLP). |
| Approach: | They propose a UNIfied-MOdal pre-training architecture which can adapt to both single-modal and multi-modal understanding and generation tasks. |
| Outcome: | The proposed model can learn more generalizable representations with rich non-paired single-modal data. |
Convolutional Neural Network for Universal Sentence Embeddings (C18-1)
Copied to clipboard
| Challenge: | Recent studies show that averaging word embeddings is effective for NLP but these models represent a sentence only in terms of features of words or uni-grams. |
| Approach: | They propose a CNN-based model that uses both features of words and n-grams to encode sentences. |
| Outcome: | The proposed model performs better than existing models in transfer learning setting and exceeds state of the art in supervised learning setting by initializing the parameters with the pre-trained sentence embeddings. |
A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning (2020.coling-main)
Copied to clipboard
| Challenge: | Paraphrase generation is of great importance for many downstream tasks in natural language processing. |
| Approach: | They propose a method to generate sentences as learning objectives from the learned data distribution and employ reinforcement learning to combine these new learning objectives for model training. |
| Outcome: | The proposed method gains significant diversity and improves generation quality over state-of-the-art datasets. |
Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoning (2025.findings-emnlp)
Copied to clipboard
Yihong Tang, Ao Qu, Zhaokai Wang, Dingyi Zhuang, Zhaofeng Wu, Wei Ma, Shenhao Wang, Yunhan Zheng, Zhan Zhao, Jinhua Zhao
| Challenge: | Currently, vision-language models excel in many downstream tasks but struggle with spatial reasoning, which is crucial for navigation and interaction with physical environments. |
| Approach: | They propose a framework that generates synthetic data to provide targeted supervision for VLMs across these basic spatial capabilities. |
| Outcome: | The proposed framework disentangles 2D spatial reasoning into three core components: direction comprehension, distance estimation, and localization. |
BKEE: Pioneering Event Extraction in the Vietnamese Language (2024.lrec-main)
Copied to clipboard
| Challenge: | Event Extraction (EE) is a fundamental task in information extraction. |
| Approach: | They propose a Vietnamese event extraction dataset that includes 33 different event types and 28 different event argument roles. |
| Outcome: | The proposed dataset provides a labeled dataset for entity mentions, event mentions and event arguments on 1066 documents. |
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Currently, large language models (LLMs) train on short text segments due to the computational overhead quadratic in the input lengths of their Transformer architectures. |
| Approach: | They propose a method that allows LLMs pre-trained with 2K or 4K-long segments to generalize to up to 200M length inputs while retaining perplexity. |
| Outcome: | The proposed method achieves 2.7 decoding speed up and 7.5 memory saving over the original model. |
Retrofitting Light-weight Language Models for Emotions using Supervised Contrastive Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | a novel retrofitting method to induce emotion aspects into pre-trained language models is proposed . the models are computationally less expensive and open, but do not capture affective aspects of human communication well. |
| Approach: | They propose a retrofitting method to induce emotion aspects into pre-trained language models . they retrofit text fragments exhibiting similar emotions into pretrained networks . |
| Outcome: | The proposed method produces emotion-aware text representations for sentiment analysis and sarcasm detection tasks. |
Wanda++: Pruning Large Language Models via Regional Gradients (2025.findings-acl)
Copied to clipboard
Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M. Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodapati, Nathan Susanj, Zheng Zhang, Jack FitzGerald, Abhishek Kumar
| Challenge: | Existing pruning methods suffer from accuracy degradation without full-model sparsity-aware fine-tuning. |
| Approach: | They propose a pruning framework that uses decoder-block-level regional gradients to improve pruning accuracy. |
| Outcome: | The proposed pruning framework outperforms the state-of-the-art pruning frameworks by utilizing decoder-block-level regional gradients. |
Enhancing Lexical Relation Mining with Structured Sememe Knowledge (2026.acl-long)
Copied to clipboard
| Challenge: | Existing top-performing methods for Lexical Relation Mining rely on pre-trained language models yet fail to distinguish nuanced lexical relations. |
| Approach: | They propose a framework to leverage structured sememe knowledge to enhance LRC and LE. |
| Outcome: | The proposed method outperforms existing methods on benchmarks and outperformed the LLMs. |
CDLM: Cross-Document Language Modeling (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing language models (LMs) provide powerful representations for internal text structure, but there are important applications for multi-text tasks. |
| Approach: | They propose a pretraining approach that incorporates two key ideas into the masked language modeling objective. |
| Outcome: | The proposed model improves over existing models and sets of long-range transformers and can be easily applied to multiple multi-text tasks. |
OKGIT: Open Knowledge Graph Link Prediction with Implicit Types (2021.findings-acl)
Copied to clipboard
| Challenge: | Open Knowledge Graphs (OpenKGs) are sparse and not directly usable in an end task. |
| Approach: | They propose a method that bootstraps OpenKGs from a corpus using OpenIE tools. |
| Outcome: | The proposed method achieves state-of-the-art performance while producing type compatible NPs in the link prediction task. |
Contextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systems (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies show that large pre-trained language models can be adapted to task-oriented dialog systems. |
| Approach: | They propose to use contextual dynamic prompting to generate prompts in dialogs . they propose to distill useful prompting signals from dialog contexts based on contextual dynamic . |
| Outcome: | The proposed approach improves response generation by 3 points and 17 points when dialog states are incorporated. |
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021.emnlp-main)
Copied to clipboard
| Challenge: | masked language models (MLMs) pre-train to model higher-order word co-occurrence statistics . authors suggest that such models have learned to represent syntactic structures prevalent in classical NLP pipelines . purely distributional information largely explains the success of pre-training, authors say . |
| Approach: | They propose to pre-train masked language models on sentences with random shuffled word order and show they still achieve high accuracy after fine-tuning on many downstream tasks. |
| Outcome: | The proposed model performs well according to parametric syntactic probes . the authors argue that the model is not all that different from earlier distributional models . |
Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models are weak and take advantage of failures and errors in datasets to improve performance. |
| Approach: | They evaluate three Transformer-based models in Natural Language Inference and Question Answering tasks to see if they are more robust or have the same flaws as their predecessors. |
| Outcome: | The proposed models outperform recurrent neural network models to stress tests on both NLI and QA tasks. |
Parameter-Efficient Fine-Tuning without Introducing New Latency (2023.acl-long)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning of pre-trained language models has been demonstrated to be effective, but its inherent characteristics limit its performance. |
| Approach: | They propose to generate a sparse mask in a task-agnostic manner by modifying only a small subset of existing parameters and adding new parameters. |
| Outcome: | The proposed method surpasses existing methods on the GLUE benchmark by a significant margin. |
When Do Pre-Training Biases Propagate to Downstream Tasks? A Case Study in Text Summarization (2023.eacl-main)
Copied to clipboard
Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, Tatsunori Hashimoto
| Challenge: | Existing studies have shown that large language models contain linguistic and societal biases, but it is unclear how these biase amplify to downstream tasks. |
| Approach: | They investigate how name-nationality bias propagates from pre-training to downstream tasks . they show that these biases manifest themselves as hallucinations in summarization . |
| Outcome: | The proposed model can reduce the rate of hallucinations, but does not change the types of biases that do appear. |
Dialogue-oriented Pre-training (2021.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PrLMs) have shown impressive improvements for various downstream tasks including various dialogue related ones. |
| Approach: | They propose to use pre-trained language models to simulate dialogue features on general plain text with common language model training objectives to improve performance. |
| Outcome: | The proposed method is fine-tuned on three public multi-turn dialogue datasets and achieves significant and consistent improvement over the plain PrLMs. |
A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Chemical named entity recognition (NER) models are used in many downstream tasks, but it is unknown whether they work the same for everyone. |
| Approach: | They develop a framework for measuring gender bias in chemical NER models . they analyze a corpus of 92,405 words with self-identified gender information from reddit . |
| Outcome: | The proposed framework measures gender bias in chemical NER models using synthetic data and a newly annotated corpus of over 92,405 words with self-identified gender information from Reddit. |
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding (2025.findings-acl)
Copied to clipboard
| Challenge: | Vision Language Models struggle with visual arithmetic, seemingly simple tasks like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. |
| Approach: | They propose a novel post-training strategy inspired by Piaget’s theory of cognitive development that trains VLMs to recognize invariant properties under visual transformations. |
| Outcome: | The proposed approach outperforms supervised fine-tuning methods while requiring 60% less training data. |
LEGAL-BERT: The Muppets straight out of Law School (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing guidelines for pre-training and fine-tuning do not always generalize well in the legal domain. |
| Approach: | They propose to use BERT out of the box, adapt it by additional pre-training on domain-specific corpora, and pre-train it from scratch on domains. |
| Outcome: | The proposed strategies are: use the original BERT out of the box, adapt it by additional pre-training on domain-specific corpora, and pre-train it from scratch on domain specific corpors. |
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pre-trained language models are ineffective, despite their potential, pre-training models suffer from important weaknesses. |
| Approach: | They analyze the extent to which the isotropy of the embedding space changes after fine-tuning. |
| Outcome: | The proposed model improves the isotropy of embedding space after fine-tuning . the model can encode linguistic properties, but lacks the social bias needed to improve it . |
Token Dropping for Efficient BERT Pretraining (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks. |
| Approach: | They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead. |
| Outcome: | The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks. |
Benchmarking Intersectional Biases in NLP (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on fairness of machine learning models has focused on how to debias, but research on the fairness and performance of biased/debiased models on downstream prediction tasks has been limited. |
| Approach: | They assess intersectional bias - fairness across multiple demographic dimensions . they highlight possible causes and make recommendations for future NLP debiasing research. |
| Outcome: | The proposed approaches fare well in terms of fairness-accuracy trade-off, but are unable to effectively alleviate bias in downstream tasks. |
When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfer (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent work on multilingual language models has demonstrated their capacity for cross-lingual zero-shot transfer on downstream tasks. |
| Approach: | They conduct a large-scale empirical study to isolate the effects of various linguistic properties by measuring zero-shot transfer between four different natural languages. |
| Outcome: | The proposed model exhibits decent cross-lingual zero-shot transfer, with no significant differences in word order and embedding alignment. |
What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks often include a mixture of reasoning questions, making it difficult to truly assess VLMs’ causal reasoning abilities. |
| Approach: | They propose two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. |
| Outcome: | The proposed benchmarks show that vision-language models perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. |
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models (2025.emnlp-main)
Copied to clipboard
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, Zheng Zhang
| Challenge: | Large Language Models (LLMs) are quantized to lower precision to reduce memory cost and latency in inference. |
| Approach: | They propose a quantized zeroth-order framework for fine-tuning Large Language Models (LLMs) using low-precision forward passes. |
| Outcome: | The proposed method achieves comparable results to first-order methods in FP8 and superior accuracy in INT8 and INT4 training. |
QANom: Question-Answer driven SRL for Nominalizations (2020.coling-main)
Copied to clipboard
Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, Ido Dagan
| Challenge: | Traditionally, SRL annotations focus on verbal predicates, but other types of predicate are frequent in natural language. |
| Approach: | They propose a semantic scheme for capturing predicate-argument relations for nominalizations, termed QANom, using crowdsourcing and QA-driven annotations. |
| Outcome: | The proposed scheme outperforms existing annotations and is useful for downstream tasks. |
A Challenge Set and Methods for Noun-Verb Ambiguity (D18-1)
Copied to clipboard
| Challenge: | English part-of-speech taggers make egregious errors related to noun-verb ambiguity, despite having achieved 97%+ accuracy on the WSJ Penn Treebank since 2002. |
| Approach: | They propose to use a WSJ dataset to identify 30,000 examples of noun-verb ambiguity . they find that english part-of-speech taggers make egregious errors related to nouns and verbs . |
| Outcome: | The proposed model improves on the WSJ Penn Treebank by 14% and 52% relative to the previous model. |
Addressing Healthcare-related Racial and LGBTQ+ Biases in Pretrained Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) propagate social stigmas and stereotypes, a critical concern given their widespread use. |
| Approach: | They adapt two intrinsic bias benchmarks to quantify racial and LGBTQ+ biases in prevalent PLMs and empirically evaluate the effectiveness of various debiasing methods in mitigating these biase. |
| Outcome: | The proposed methods reduce biases without compromising performance in downstream tasks. |
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024.findings-naacl)
Copied to clipboard
| Challenge: | Comparative knowledge is an essential component of our world knowledge, yet understudied in prior literature. |
| Approach: | They propose a framework for comparative knowledge distillation overgenerated from language models . they use a corpus of 8.8M comparisons over 1.74M entity pairs to acquire comparative information . |
| Outcome: | The proposed framework acquires comparative knowledge between everyday objects . human evaluations show that it outperforms existing resources in terms of validity . |
Task-oriented Domain-specific Meta-Embedding for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods neglect domain-specific knowledge and use the same word embedding for each word in all domain-specified datasets. |
| Approach: | They propose a method to incorporate domain-specific and task-oriented information into meta-embeddings by combining pre-trained word embeddings. |
| Outcome: | The proposed method performs well on four text classification datasets and shows that it is compatible with existing methods. |
On Utilizing Constituent Language Resources to Improve Downstream Tasks in Hinglish (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Intuitively, Hindi and English corpora should aid improve task performance on code-switched Hindi-English. |
| Approach: | They propose a meta-learning framework that utilizes the labelled resources of the downstream tasks in the constituent languages to improve task performance. |
| Outcome: | The proposed framework improves the performance on downstream tasks on code-switched Hindi-English. |
Open Information Extraction on Scientific Text: An Evaluation (C18-1)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) is the unsupervised creation of structured information from text. |
| Approach: | They propose to use two state-of-the-art OIE systems to evaluate the performance of OIE on scientific texts originating from 10 different disciplines. |
| Outcome: | The proposed methods perform significantly worse on scientific text than encyclopedic text. |
FlauBERT: Unsupervised Language Model Pre-training for French (2020.lrec-1)
Copied to clipboard
Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, Didier Schwab
| Challenge: | Language models are a key step to achieve state-of-the-art results in many different Natural Language Processing (NLP) tasks. |
| Approach: | They propose to use a language model that is pre-trained on a large and heterogeneous French corpus to train continuous word representations. |
| Outcome: | The proposed model outperforms existing models on a large and heterogeneous French corpus. |
Constraining Sequential Model Editing with Editing Anchor Compression (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) exhibit hallucinations due to incorrect or outdated knowledge embedded in their parameters. |
| Approach: | They propose a framework to constrain the deviation of the parameter matrix during sequential editing by selecting editing anchors that are important in encoding new relations without deviating too much from the original matrix. |
| Outcome: | The proposed framework minimizes deviations caused by model editing while retaining over 70% of the general abilities. |
Learning multiview embeddings for assessing dementia (D18-1)
Copied to clipboard
| Challenge: | In 2017, 5.7 million Americans were living with Alzheimer's disease (AD), and the disease accounted for $11.4 billion in healthcare costs in the United States. |
| Approach: | They leverage the multiview nature of a small AD dataset to learn an embedding that captures different modes of cognitive impairment. |
| Outcome: | The proposed embeddings achieve an F1 score of 0.82 and a mean absolute error of 3.42 in the classification task and predicting clinical scores. |
Using RL to Identify Divisive Perspectives Improves LLMs Abilities to Identify Communities on Social Media (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show improvements on Reddit and Twitter data . |
| Approach: | They propose to take advantage of Large Language Models (LLMs) to better identify user communities. |
| Outcome: | The proposed model improves on Reddit and Twitter data and tasks of community detection, bot detection, and news media profiling. |
Mining Knowledge for Natural Language Inference from Wikipedia Categories (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Accurate lexical entailment (LE) and natural language inference (NLI) tasks require expensive annotations. |
| Approach: | They propose to pretrain Wikipedia categories for lexical entailment and natural language inference by pretraining them on WikiNLI and transferring them to other knowledge bases. |
| Outcome: | The proposed model can improve strong baselines such as BERT and RoBERTa by pretraining on WikiNLI and transferring the models on downstream tasks. |
Slender-Mamba: Fully Quantized Mamba in 1.58 Bits From Head to Toe (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved significant performance improvements in natural language processing domain, but require large computational resources for training and inference. |
| Approach: | They propose to use a language model architecture based on State-Space Models to quantify embedding and projection layers of a model with 150 B tokens from scratch. |
| Outcome: | The proposed language model architecture reduces costs by compressing context windows during inference while reducing the cost of training and inference. |
NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian (2024.emnlp-main)
Copied to clipboard
Peng Liu, Lemei Zhang, Terje Farup, Even Lauvrak, Jon Ingvaldsen, Simen Eide, Jon Atle Gulla, Zhirong Yang
| Challenge: | Norwegian is under-represented within the most impressive breakthroughs in NLP tasks. |
| Approach: | they investigate the impact of existing Norwegian language models on Norwegian generation tasks . they pre-trained 4 Norwegian Open Language Models from parameter scales and architectures . |
| Outcome: | The proposed benchmark evaluates the performance of language models on Norwegian generation tasks. |
Query Rewriting in Retrieval-Augmented Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on adapting either the retriever or the reader, but this approach is more focused on adaptation of the query itself. |
| Approach: | They propose a new framework for retrieval-augmented Large Language Models . they propose rewrite-retrieve-read instead of retrieve-then-read . |
| Outcome: | The proposed framework improves performance on downstream tasks, open-domain QA and multiple-choice QA. |
Parameter-Efficient Domain Knowledge Integration from Multiple Sources for Biomedical Pre-trained Language Models (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing domain-specific pre-trained language models (PLMs) rely on self-supervised learning over large amounts of domain text, without explicitly integrating domain- specific knowledge. |
| Approach: | They propose to integrate domain knowledge from diverse sources into PLMs by using adapters that are pre-trained for individual domain knowledge sources and integrated via an attention-based knowledge controller. |
| Outcome: | The proposed architecture integrates domain knowledge from diverse sources into PLMs in a parameter-efficient way. |
PEMA: An Offsite-Tunable Plug-in External Memory Adaptation for Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) show impressive performance in various downstream NLP tasks. |
| Approach: | They propose a Parameter-Efficient Fine-Tuning method that integrates with context representations from test data to perform downstream tasks. |
| Outcome: | The proposed method outperforms other methods in memory and latency efficiency and maintains sentence meaning and generating appropriate language and styles. |
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating scientific document representations fail to capture the diversity of relevant tasks. |
| Approach: | They propose a benchmark for training and evaluating scientific document representations that includes 24 challenging and realistic tasks across four formats: classification, regression, ranking and search. |
| Outcome: | The proposed model outperforms existing models by over 2 points absolute. |
DSP: Discriminative Soft Prompts for Zero-Shot Entity and Relation Extraction (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompt-based methods have shown their efficacy in transferring general knowledge within pre-trained language models (PLMs) however, when applied to zero-shot entity and relation extraction, they struggle with the limited coverage of verbalizers to labels and the slow inference speed. |
| Approach: | They propose a method which reformulates zero-shot tasks into token discrimination tasks without having to construct verbalizers. |
| Outcome: | The proposed method outperforms baselines on two zero-shot entity recognition datasets with higher inference speed and achieves 7.5% improvement over previous state-of-the-art models on Wiki-ZSL and FewRel. |
Measuring the Instability of Fine-Tuning (2023.acl-long)
Copied to clipboard
| Challenge: | Many previous studies have investigated fine-tuning pre-trained language models on downstream tasks with varying random seeds, but they only used the standard deviation of performance scores (SD) as their measure, which is a narrow characterization of instability. |
| Approach: | They propose a systematic evaluation framework for the standard deviation of performance scores (SD) and six other measures quantifying instability of different granularity levels. |
| Outcome: | The proposed framework will be used to evaluate the validity of these measures and to improve them. |
Singlish Message Paraphrasing: A Joint Task of Creole Translation and Text Normalization (2022.coling-1)
Copied to clipboard
| Challenge: | Existing computational approaches to translate languages or creoles back to standard English are challenging . lexical level normalization, syntactic level editing, and semantic level rewriting are key to a successful translation task. |
| Approach: | They propose a computational task to parse Singlish into English using its dialects . they propose to use a dataset to normalize and edit the text to improve translation . |
| Outcome: | The proposed model can improve translation performance and improve stance detection. |
Tracking Life’s Ups and Downs: Mining Life Events from Social Media Posts for Mental Health Analysis (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have indicated that major life events can greatly impact individuals’ mental health, but shedding its light on social media data is challenging due to the complexity and ambiguity nature of life events. |
| Approach: | They propose to extract life events mentioned in posts on social media to uncover a social media event dataset which includes 12 major life event categories that are likely to occur in everyday life. |
| Outcome: | The proposed dataset includes 12 life event categories that are likely to occur in everyday life and is human-annotated under iterative procedure and boasts a high level of quality. |
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained language models can be more efficient when they are larger than they are in their size. |
| Approach: | They propose a prompt-based transfer learning approach called SPoT: Soft Prompt Transfer that learns a soft prompt on one or more source tasks and initializes it for a target task. |
| Outcome: | The proposed approach outperforms Prompt Tuning and MODELTUNING on superGLUE benchmarks while using up to 27,000 fewer task-specific parameters. |
Pass off Fish Eyes for Pearls: Attacking Model Selection of Pre-trained Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing feature-based model selection methods are vulnerable to fine-tuning and are not reliable indicators for the PTM’s transferability. |
| Approach: | They propose feature-based model selection methods which assess PTMs’ transferability to a specific task in a fast way without fine-tuning. |
| Outcome: | The proposed methods can make FMS mistakenly judge transferability of models and can be combined with the backdoor attack to misguide the FMS to select poisoned models. |
GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models (2026.acl-long)
Copied to clipboard
Zhiwen Ruan, Yichao Du, Jianjie Zheng, Longyue Wang, Yun Chen, Peng Li, Jinsong Su, Yang Liu, Guanhua Chen
| Challenge: | Existing adapter-based transfer methods treat instruction-tuned models as passive targets . direct fine-tuning can disrupt this delicate balance and lead to instability or performance degradation. |
| Approach: | They propose a framework that incorporates instruction-level guidance into task adaptation. |
| Outcome: | The proposed framework outperforms direct fine-tuning and representative transfer-based baselines while maintaining robust generalization and favorable test-time scaling behavior. |
HEAL: Hybrid Enhancement with LLM-based Agents for Text-attributed Hypergraph Self-supervised Representation Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to enhance text-attributed hypergraph self-supervised learning are limited by label scarcity. |
| Approach: | They propose a data-centric approach that leverages large language models to enhance hypergraph self-supervised learning by integrating hyperedges into a self-representation framework. |
| Outcome: | The proposed approach generates informative nodes and hyperedges through multi-round interaction with LLM-based agents. |
A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretraining languages improve cross-lingual transfer for BERT-based models . Interestingly, PLMs exhibit zero-shot cross-linguistic abilities on downstream examples in languages seen only during pretraining. |
| Approach: | They develop a quadratic time complexity method to estimate pretraining languages' relations between linguistic features and two downstream tasks. |
| Outcome: | The proposed method is effective on a diverse set of languages spanning different linguistic features and two downstream tasks. |
AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing MoE methods require a constant top-k routing for all tokens, which is restrictive because of the number of experts required for feature abstraction. |
| Approach: | They propose a token-adaptive routing method that allows different tokens to select a different number of experts. |
| Outcome: | a new method can reduce average expert load while achieving superior performance. |
Emotion Enriched Retrofitted Word Embeddings (2022.coling-1)
Copied to clipboard
| Challenge: | Word embeddings that encode lexical-semantic relations do not capture emotion aspects of words. |
| Approach: | They propose a retrofitting method to update the vectors of emotion bearing words . they find that the retrofitted embeddings achieve better distances between clusters . |
| Outcome: | The proposed method achieves better distances between clusters and clusters for words having the same emotions. |
Detection and Measurement of Syntactic Templates in Generated Text (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing diversity evaluation focuses primarily on word-level features. |
| Approach: | They propose a method for evaluating diversity over syntactic features to characterize general repetition in large language models. |
| Outcome: | The proposed method shows that models produce templated text in downstream tasks at a higher rate than what is found in human-reference texts. |
Commonsense Knowledge Transfer for Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in pre-trained language models have transformed the landscape of natural language processing. |
| Approach: | They propose a framework to transfer commonsense knowledge stored in a neural commonsensing model to a general-purpose pre-trained language model. |
| Outcome: | Empirical results show that the proposed framework improves the model’s performance on downstream tasks that require commonsense reasoning. |
GoEmotions: A Dataset of Fine-Grained Emotions (2020.acl-main)
Copied to clipboard
| Challenge: | Existing datasets for language-based emotion classification are limited and small . existing datasets lack quality annotations for many different emotion categories . |
| Approach: | They propose to use a large manually annotated dataset to study emotion expressions . they conduct transfer learning experiments with existing emotion benchmarks to test their model . |
| Outcome: | The proposed model achieves an average F1-score of .46, leaving room for improvement. |
Text Segmentation by Cross Segment Attention (2020.emnlp-main)
Copied to clipboard
| Challenge: | Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents. |
| Approach: | They propose three transformer-based NLP models that break up text into constituents and compare them to previous approaches. |
| Outcome: | The proposed architectures reduce errors by a large margin on three datasets and improve performance on real-world datasets. |
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT (2021.emnlp-main)
Copied to clipboard
| Challenge: | Infusing factual knowledge into pre-trained models is fundamental for many knowledge-intensive tasks. |
| Approach: | They propose an infusion approach that partitions a large knowledge graph into smaller sub-graphs and infuses their specific knowledge into various BERT models using lightweight adapters. |
| Outcome: | The proposed approach improves the underlying BERTs and achieves new SOTA performance on six downstream tasks. |
Multi-Relational Probabilistic Event Representation Learning via Projected Gaussian Embedding (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for event representation learning ignore relations and uncertainty of events . Experimental results show that the proposed approach outperforms other state-of-the-art baselines on both existing and newly constructed datasets. |
| Approach: | They propose a novel approach to learning multi-relational probabilistic event embeddings based on contrastive learning. |
| Outcome: | The proposed method outperforms existing benchmarks on existing and newly constructed datasets. |
Methods for Numeracy-Preserving Word Embeddings (2020.emnlp-main)
Copied to clipboard
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, Lawrence Carin
| Challenge: | Word embedding models capture semantic relationships between words but fail to capture numerical properties associated with numbers. |
| Approach: | They propose a method to assign and learn embeddings for numbers using word embedders. |
| Outcome: | The proposed model outperforms pre-trained word embedding models across multiple examples of two tasks. |
Patton: Language Model Pretraining on Text-Rich Networks (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for text-rich networks do not take inter-document structure into account. |
| Approach: | They propose a pretraining framework for a text-rich network using a masked language model and a masking node prediction framework. |
| Outcome: | The proposed model outperforms baselines on four tasks in academic and e-commerce domains. |
Multi-target Backdoor Attacks for Code Pre-trained Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work for backdoor attacks on neural code models insert triggers into task-specific data for code-related downstream tasks, limiting the scope of attacks. |
| Approach: | They propose task-agnostic backdoor attacks for code pre-trained models . they use two learning strategies to implant backdoors into code understanding and generation models - Poisoned Seq2Seq learning and token representation learning . |
| Outcome: | The proposed model is pre-trained with two learning strategies to support the multi-target attack of downstream code understanding and generation tasks. |
Don’t Judge a Language Model by Its Last Layer: Contrastive Learning with Layer-Wise Attention Pooling (2022.coling-1)
Copied to clipboard
| Challenge: | Recent pre-trained language models (PLMs) have shown competitive performance on many natural language processing tasks. |
| Approach: | They propose a pooling strategy which preserves layer-wise signals captured in each layer and learns digested linguistic features for downstream tasks. |
| Outcome: | The proposed method improves on standard semantic textual similarity and semantic search tasks. |
HIT - A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation (2021.findings-acl)
Copied to clipboard
| Challenge: | linguistics and morphology of resource-short code-mixed texts remain a key challenge in text processing. |
| Approach: | They propose a hierarchical transformer-based framework that captures the semantic relationship among words and hierarchically learns sentencelevel semantics using a fused attention mechanism. |
| Outcome: | The proposed framework improves on one European and five Indic languages on four NLP tasks on eleven datasets. |
Counterfactual Adversarial Learning with Representation Interpolation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models with statistical bias are prone to memorized correlations . large pre-trained models such as BERT have revolutionized the model development paradigm in natural language processing . |
| Approach: | They propose a framework to tackle the problem from a causal perspective using a latent space interpolation approach. |
| Outcome: | Extensive experiments show that CAT achieves substantial performance improvement over SOTA across different downstream tasks, including sentence classification, natural language inference and question answering. |
Lexically Grounded Subword Segmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Statistical word segmentation algorithms have remained a thorn in the side of many researchers. |
| Approach: | They propose to use unsupervised morphological analysis with Morfessor as pre-tokenization and an algebraic method for obtaining subword embeddings grounded in a word embeddable space. |
| Outcome: | The proposed methods improve morphological plausibility and Rényi efficiency on part-of-speech tagging and machine translation tasks. |
MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2022.acl-long)
Copied to clipboard
| Challenge: | Prompting has been shown to be a promising approach for applying pre-trained language models to perform downstream tasks. |
| Approach: | They propose a method that divides the translation process into three stages using pre-trained language models. |
| Outcome: | The proposed method significantly improves translation performance of pre-trained language models on three translation tasks. |
ModaFact: Multi-paradigm Evaluation for Joint Event Modality and Factuality Detection (2025.coling-main)
Copied to clipboard
| Challenge: | NLP studies have mostly dealt with factuality and modality separately . linguistic modality conveys the relationship a situation is supposed to have with respect to wishes, norms, goals, authority, etc. |
| Approach: | They propose a resource with joint factuality and modality information for event-denoting expressions in Italian. |
| Outcome: | The proposed resource is consistent with existing ones and compares classification systems trained on italy's ModaFact dataset and best-performing model. |
Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to improve neural language models perform poorly on emerging data. |
| Approach: | They propose a lexical-level masking strategy to post-train a neural language model using static data from past years. |
| Outcome: | The proposed method outperforms existing methods on two pre-trained language models, two classification tasks, and four benchmark datasets. |
How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work probing pre-trained language models for downstream tasks is difficult to explain . a growing body of research is devoted to understanding what linguistic properties these language models have acquired. |
| Approach: | They propose a procedure and analysis method that takes a hypothesis of how a transformer-based model might encode a linguistic phenomenon and tests its validity. |
| Outcome: | The proposed method tests a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. |
MUZO: Leveraging Multiple Queries and Momentum for Zeroth-Order Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning large language models incur memory overhead due to the need for activation storage for back-propagation (BP). |
| Approach: | They propose a method that estimates gradients through finite differences without activation storage for back-propagation. |
| Outcome: | The proposed method demonstrates superior performance in fine-tuning various LLMs. |
Robust Backed-off Estimation of Out-of-Vocabulary Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to solving out-of-vocabulary (OOV) words use subwords to represent oov words with a bag of subword. |
| Approach: | They propose a method to estimate oov word embeddings by referring to pre-trained word embeds for known words with similar surfaces to target ov words. |
| Outcome: | The proposed method improves word similarity tasks and biomedical tasks even with weak baselines. |
RotateCT: Knowledge Graph Embedding by Rotation and Coordinate Transformation in Complex Space (2022.coling-1)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding methods fail to model non-commutative composition patterns . Existing methods are limited to complex space, resulting in a large number of parameters. |
| Approach: | They propose a knowledge graph embedding method that transforms the coordinates of each entity and then represents each relation as a rotation from head entity to tail entity in complex space. |
| Outcome: | The proposed method outperforms state-of-the-art methods on link prediction and path query answering. |
Title2Event: Benchmarking Open Event Extraction with a Large-scale Chinese Title Dataset (2022.emnlp-main)
Copied to clipboard
Haolin Deng, Yanan Zhang, Yangfan Zhang, Wangyang Ying, Changlong Yu, Jun Gao, Wei Wang, Xiaoling Bai, Nan Yang, Jin Ma, Xiang Chen, Tianhua Zhou
| Challenge: | Existing EE datasets define fixed event types and design specific schemas for each of them, failing to cover diverse events emerging from the online text. |
| Approach: | They propose to use a sentence-level dataset to benchmark Open Event Extraction without restricting event types. |
| Outcome: | The proposed dataset contains more than 42,000 news titles in 34 topics collected from Chinese web pages. |
How transfer learning impacts linguistic knowledge in deep NLP models? (2021.findings-acl)
Copied to clipboard
| Challenge: | Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. |
| Approach: | They propose to fine-tune pre-trained models towards downstream NLP tasks to capture linguistic knowledge. |
| Outcome: | The proposed model is adapted to GLUE tasks and retains linguistic information in the network while forgetting it. |
Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameters (2022.emnlp-main)
Copied to clipboard
| Challenge: | Adapter-tuning is a paradigm that transfers a pretrained language model to downstream tasks . Previously proposed adapters are all feed-forward neural networks . |
| Approach: | They propose to use tiny-attention attention with extremely small per-head dimensionality as adapters to modify hidden states at each position . they propose to average multiple attention heads' weights during deployment to reduce its inference computation cost. |
| Outcome: | The proposed adapter outperforms other adapter-tuning methods on the GLUE benchmark . it uses attention with extremely small per-head dimensionality to modify hidden states . |
Monolingual and Multilingual Reduction of Gender Bias in Contextualized Representations (2020.coling-main)
Copied to clipboard
| Challenge: | Prior work identifies a linear gender subspace and removes gender information by eliminating the subspace. |
| Approach: | They propose to use DensRay to obtain interpretable dense subspaces by applying it to attention heads and layers of BERT. |
| Outcome: | The proposed method performs on-par with prior approaches, but is more robust and preserves language model performance better. |
Comparing Prompt-Based and Standard Fine-Tuning for Urdu Text Classification (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have demonstrated the efficacy of pre-trained language models for various downstream tasks. |
| Approach: | They compare prompt-based fine-tuning with standard fine-uning for text classification in Urdu and Roman Urdu languages. |
| Outcome: | The proposed approach improves up to 13% in accuracy in low-resource languages with limited labeled examples over standard fine-tuning approaches. |
Vector-Quantized Input-Contextualized Soft Prompts for Natural Language Understanding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Prompt Tuning has been successful as a parameter-efficient method of conditioning large-scale pre-trained language models to perform downstream tasks. |
| Approach: | They propose to use a vector-quantized input-contextualized prompt as an extension to the soft prompt tuning framework to learn contextualization of soft prompt tokens. |
| Outcome: | The proposed prompt outperforms soft prompt tuning by an average margin of 1.19% on various language understanding tasks like SuperGLUE, QA, Relation classification, NER and NLI. |
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English. |
| Approach: | They propose a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols. |
| Outcome: | The proposed method defends against inflectional adversaries while maintaining performance on clean data. |
Mitigating Negative Interference in Multilingual Knowledge Editing through Null-Space Constraints (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing monolingual knowledge editing methods are expensive and require multiple models to maintain factual consistency. |
| Approach: | They propose a null-space constrained framework to precisely isolate language-specific knowledge updates that can be mapped onto other languages’ subspaces. |
| Outcome: | The proposed framework can project parameter updates for each language onto the orthogonal complement of other languages’ subspaces while preserving multilingual generalization capabilities. |
Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have shown strong potential in various downstream tasks. |
| Approach: | They propose to model adversarial attack task as a sequential decision-making problem where the whole attack process is sequential with two decision- making problems, i.e., word finder and word substitution. |
| Outcome: | The proposed approach achieves the highest attack success rate with a comparable modification rate and semantic similarity to attack fine-tuned BERT. |
A Neural Local Coherence Model for Text Quality Assessment (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to local coherence modeling capture text relatedness at the level of sentence-to-sentence transitions. |
| Approach: | They propose a local coherence model that captures the flow of what connects adjacent sentences . they represent the semantics of a sentence by a vector and capture its state at each word . |
| Outcome: | The proposed model is beneficial for readability assessment and essay scoring tasks. |
Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to pretrain language models are limited by one-size-fits-all vocabulary . embeddings of mismatch tokens can be efficiently initialized in downstream tasks . |
| Approach: | They propose to extend pretrain-finetune pipeline with an embedding transfer step . plug-and-play embeddable generator is introduced to generate any input token . |
| Outcome: | The proposed approach allows for more efficient and better performed NLG models. |
UOR: Universal Backdoor Attacks on Pre-trained Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to attack pre-trained language models rely on manual selection of triggers and backdoor representations. |
| Approach: | They propose a backdoor attack method that turns manual selection into automatic optimization . they propose to use poisoned contrastive learning to learn more uniform backdoor representations . |
| Outcome: | The proposed method achieves better attack performance on text classification tasks compared to manual methods. |
Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP (2021.emnlp-main)
Copied to clipboard
| Challenge: | Meta-learning considers learning as an efficient learning process that can leverage its past experience to accurately solve new tasks. |
| Approach: | They propose to provide task distributions for meta-learning by considering self-supervised tasks automatically proposed from unlabeled text to enable large-scale meta- learning in NLP. |
| Outcome: | The proposed distributions show that human learning models perform better on the few-shot benchmark than previous methods. |
Are Visual-Linguistic Models Commonsense Knowledge Bases? (2022.coling-1)
Copied to clipboard
| Challenge: | PTLMs are used to extract knowledge from text on demand. |
| Approach: | They compare visual-linguistic and language-only visual-language models in a zero-shot commonsense question answering inference task. |
| Outcome: | The proposed models are highly promising on certain types of commonsense knowledge associated with the visual world. |
On Tree-Based Neural Sentence Modeling (D18-1)
Copied to clipboard
| Challenge: | Existing tree-based sentence modeling approaches adopt syntactic parsing trees as the explicit structure prior. |
| Approach: | They replace parsing trees with trivial trees to study their effectiveness . they found that tree-based sentence modeling gives better results when crucial words are closer to the final representation . |
| Outcome: | The proposed tree-based sentences have shown better results on many downstream tasks. |
Event Representation Learning Enhanced with External Commonsense Knowledge (D19-1)
Copied to clipboard
| Challenge: | Existing methods to learn event representations from text lack commonsense knowledge about the intents and emotions of event participants. |
| Approach: | They propose to leverage external commonsense knowledge about the intent and sentiment of the event to learn distributed representations for structured events from text. |
| Outcome: | The proposed model improves on hard similarity tasks and yields more precise inferences on subsequent events under given contexts. |
Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that pretraining with an artificial language with nesting dependency structure provides some knowledge transferable to natural language. |
| Approach: | They propose to pretrain artificial languages with structural properties that mimic natural language and then test their performance on downstream tasks. |
| Outcome: | The proposed language models show strong performance across languages and languages. |
Deeply Coupled Cross-Modal Prompt Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing prompt-tuning methods focus on language branch or learn vision-language interaction in a shallow mechanism. |
| Approach: | They propose a Deeply coupled Cross-modal Prompt learning method based on CLIP to facilitate the interplay between vision and language with a Cross-Modal Prompting Attention mechanism. |
| Outcome: | The proposed method enables the interplay between vision and language with a Cross-Modal Prompt Attention mechanism. |
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding
| Challenge: | Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks. |
| Approach: | They propose a mix-of-experts model that allows the model size to grow without raising training costs. |
| Outcome: | The proposed model outperforms existing models in perplexity and robustness tests. |
NNE: A Dataset for Nested Named Entity Recognition in English Newswire (P19-1)
Copied to clipboard
| Challenge: | Named entity recognition (NER) is widely used in downstream tasks but most tools focus on flat mention structure over coarse schemas. |
| Approach: | They describe a fine-grained, nested named entity dataset over the Wall Street Journal portion of the Penn Treebank. |
| Outcome: | The proposed dataset comprises 279,795 mentions of 114 entity types with up to 6 layers of nesting. |
TransLIST: A Transformer-Based Linguistically Informed Sanskrit Tokenizer (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to SWS fail when encountering out-of-vocabulary tokens . lexicon driven approaches fail when dealing with out- of-vocal tokens, authors say . |
| Approach: | They propose a novel lexicon-driven shallow parser that takes into account sandhi phenomenon . they propose linguistically informed Sanskrit tokenizer module that encodes character input . |
| Outcome: | The proposed system outperforms the current state-of-the-art system by 7.2 points absolute gain on the benchmark datasets. |
Enhancing Automatic Term Extraction with Large Language Models via Syntactic Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have improved IE, but their potential for ATE has not been explored. |
| Approach: | They propose a retrieval-based prompting strategy that selects demonstrations according to syntactic rather than semantic similarity in a few-shot setting. |
| Outcome: | The proposed method improves performance on three specialized ATE benchmarks. |
Scalable and Domain-General Abstractive Proposition Segmentation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Several recent studies have demonstrated the utility of proposition segmentation for downstream tasks. |
| Approach: | They propose a scalable, yet accurate, proposition segmentation model that can be supervised by LLMs. |
| Outcome: | The proposed model improves on training on annotated datasets and shows that it is easy to use. |
Hit the Nail on the Head: Parameter-Efficient Multi-task Tuning via Human Language Intervention (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that PEFT on small pre-trained language models improves multitasking capabilities. |
| Approach: | They propose a multi-task learning framework that enables transfer of prior knowledge across tasks . they attach task descriptions to input samples and map them to task embeddings . |
| Outcome: | The proposed method improves performance on a T5 model and in decoder-only models . |
Learning Dense Representations of Phrases at Scale (2021.acl-long)
Copied to clipboard
| Challenge: | Existing phrase retrieval models rely on sparse representations and still underperform retriever-reader approaches. |
| Approach: | They propose a method to learn phrase representations from reading comprehension tasks using negative sampling methods. |
| Outcome: | The proposed model improves over previous models by 15%-25% absolute accuracy and matches the performance of state-of-the-art retrieval models. |
Achieving Stronger Generation via Simple Contrastive Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years have witnessed remarkable progress in large language models (LLMs). |
| Approach: | They propose a framework for contrastive decoding to enhance instruction-tuned models. |
| Outcome: | The proposed framework improves model performance without additional data or computational resources. |
LoRMA: Low-Rank Multiplicative Adaptation for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive generalization capabilities, but can be expensive to fine-tune due to high computational costs. |
| Approach: | They propose a low-rank multiplicative Adaptation technique that shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. |
| Outcome: | The proposed approach overcomes computational complexity and rank bottlenecks in terms of matrix multiplication metrics. |
QASem Parsing: Text-to-text Modeling of QA-based Semantics (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing work suggests the appeals of incorporating explicit semantic representations into NLP . semi-structured natural language structures provide an intermediate meaning-capturing representation . |
| Approach: | They propose a semi-structured natural-language representation of textual information . they examine input and output linearization strategies and multitask learning . |
| Outcome: | The proposed model is based on pre-trained sequence-to-sequence language models . it is easy to use and can be used for downstream tasks that benefit from it . |
A Prism Module for Semantic Disentanglement in Name Entity Recognition (P19-1)
Copied to clipboard
| Challenge: | Xu et al., 2015) proposed a noise reduction mechanism to disentangle semantics of words . hard and soft attention mechanisms are used to reduce noise in NLP tasks . |
| Approach: | They propose a prism module to disentangle semantic aspects of words and reduce noise . they propose combining prism modules with downstream models to improve model performance . |
| Outcome: | The proposed method significantly improves the performance of baselines on named entity recognition (NER) tasks. |
Enhancing Parameter-efficient Fine-tuning with Simple Calibration Based on Stable Rank (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for lightweight fine-tuning are ineffective in low-resource settings but fail in high-resourced settings, leading to unreliable outcomes. |
| Approach: | They propose a calibration strategy that takes into account the inherent variance of generalization ability in model components and potential changes during the fine-tuning process. |
| Outcome: | The proposed calibration improves GLUE score by 3.1 points over the previous calibration method. |
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent large-scale video-language pre-trained models have shown appealing performance on downstream tasks. |
| Approach: | They propose a video-text model that adapts a pre-trained image-language model into a text-based model without heavy pre-training. |
| Outcome: | The proposed model outperforms existing models on video-text retrieval and video question answering tasks without heavy pre-training. |
A Free/Open-Source Morphological Analyser and Generator for Sakha (2022.lrec-1)
Copied to clipboard
| Challenge: | a morphological transducer for Sakha is being developed for use in downstream tasks . the marginalised language is subject to increasing economic and cultural peril due to climate change . |
| Approach: | They describe the development of a morphological analyser and generator for Sakha . the transducer has coverage of solidly above 90%, and high precision . it is already being used in downstream tasks such as linguistic maintenance . |
| Outcome: | The proposed morphological analyser has coverage of 90% and high precision . it is already being used in computer assisted language learning applications . |
LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language model pretraining methods do not capture dependencies or knowledge that span across documents. |
| Approach: | They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs . |
| Outcome: | The proposed method outperforms existing methods on downstream tasks across two domains. |
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to build a strong multilingual multimodal representation model are lacking in good-quality text-image pairs. |
| Approach: | They propose a method to build a strong multilingual multimodal representation model using English text-image pairs instead of a model from scratch. |
| Outcome: | The proposed model outperforms the original CLIP model on multilingual multimodal benchmarks. |
Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing supervised sentence embedding techniques rely on expensive human-annotated sentence pairs as the supervised signals. |
| Approach: | They propose a semi-supervised sentence embedding framework that leverages large-scale unlabeled data. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on four domain adaptation tasks. |
Contrastive Deterministic Autoencoders For Language Modeling (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Variational autoencoders (VAEs) are a popular family of generative models with wide applicability. |
| Approach: | They propose to modify a deterministic model designed for images to avoid posterior collapse by controlling the entropy of the aggregate posterior to make it Gaussian. |
| Outcome: | The proposed models outperform a broad range of VAE models on text generation and downstream tasks from representations while avoiding reparametrization steps. |
Stimulate the Critical Thinking of LLMs via Debiasing Discussion (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that large language models (LLMs) are often prone to stance homogeneity and human preference biases when faced with conflicting perspectives. |
| Approach: | They propose a novel two-stage training framework to address stance homogeneity bias and human preference bias by generating multi-model discussion datasets and optimizing reinforcement learning from human feedback to align with discussion correctness. |
| Outcome: | The proposed framework reduces stance homogeneity bias and human preference bias and improves generalization capabilities on non-discussion scenarios and out-of-domain datasets. |
Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings (2024.lrec-main)
Copied to clipboard
| Challenge: | Sentence embeddings produced by pretrained language models are high dimensional (ca. 1024-4096) this is problematic when representing large numbers of sentences in memory- or compute-constrained devices. |
| Approach: | They propose to use Principal Component Analysis to reduce the dimensionality of sentence embeddings produced by pretrained language models to reduce their complexity. |
| Outcome: | The proposed methods reduce the dimensionality of sentence embeddings by 50% without incurring significant loss in performance in multiple downstream tasks. |
Revisiting Token Dropping Strategy in Efficient BERT Pretraining (2023.acl-long)
Copied to clipboard
| Challenge: | Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT. |
| Approach: | They propose a semantic-consistent learning method to improve token dropping by skipping the computation of a subset of input tokens at several middle layers. |
| Outcome: | The proposed method achieves consistent and significant performance gains across all tasks and model sizes. |
Optimal Transport-based Alignment of Learned Character Representations for String Similarity (P19-1)
Copied to clipboard
| Challenge: | String similarity models are crucial for record linkage, data integration, search and entity resolution systems. |
| Approach: | They propose a model that encodes the characters of each string, aligns the encodings using Sinkhorn Iteration and scores the alignment with a convolutional neural network. |
| Outcome: | The proposed model outperforms state-of-the-art and classical similarity models on four of the five datasets and improves performance by applying it to cross-document coreference. |
Word Embedding Evaluation in Downstream Tasks and Semantic Analogies (2020.lrec-1)
Copied to clipboard
| Challenge: | Language Models (LMs) are an oft studied area of natural language processing . Word Embeddings (WE) are vector space representations of a vocabulary . |
| Approach: | They evaluate Word Embeddings (WE) models for the Portuguese langauage . results show that a diverse corpus can often outperform a larger, less textually diverse corp. |
| Outcome: | The proposed models outperform a larger, less textually diverse corpus in two tasks . the evaluation shows that a diverse and comprehensive corpus outperformed a smaller, less diverse corp. |
Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a key-value cache, memory consumption is a bottleneck for high-throughput language models. |
| Approach: | They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput. |
| Outcome: | The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques. |
Text Rendering Strategies for Pixel Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent approaches to rendering text use a large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks due to redundancy in the input representations. |
| Approach: | They propose four approaches to rendering text in a PIXEL model using character bigrams and patch frequency biases. |
| Outcome: | The proposed models perform better on sentence-level tasks without compromising performance on token-level or multilingual tasks. |
PatchBERT: Just-in-Time, Out-of-Vocabulary Patching (2020.emnlp-main)
Copied to clipboard
| Challenge: | a pre-trained language model with low OOV can improve performance for transfer learning . a vocabulary surrogate can provide performance boosts with no additional computation cost . |
| Approach: | They propose multiple methods to mitigate OOV during downstream task fine-tuning . they demonstrate that vocabulary surrogates can provide performance boosts with no additional computation cost . |
| Outcome: | The proposed methods improve performance with the same parameter count when combined with fine-tuning. |
Exploring Graph Representations of Logical Forms for Language Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | Graph-based formal-logical distributional semantics models are more data-efficient than textual counterparts. |
| Approach: | They propose a pretrained language model over graph representations of logical forms as a proof-of-concept. |
| Outcome: | The proposed model outperforms textual, transformer LMs on downstream tasks . the model is likely to scale with additional parameters and pretraining data . |
The importance of fillers for text representations of speech transcripts (2020.emnlp-main)
Copied to clipboard
| Challenge: | Fillers are a type of disfluency that can be a sound ("um" or "uh") filling a pause in an utterance or conversation. |
| Approach: | They propose to represent fillers with deep contextualised embeddings to improve modelling of spoken language and two downstream tasks . |
| Outcome: | The proposed representations improve modelling of spoken language and two downstream tasks, predicting a speaker’s stance and expressed confidence. |
Revisiting Pruning vs Quantization for Small Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Compressing Small Language Models (SLMs) is particularly suited for resource-constrained devices, but their compression dynamics remain underexplored compared to Large Language Model (LLMs). |
| Approach: | They evaluated post-training pruning and quantization methods across six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks. |
| Outcome: | The proposed methods outperform pruning and quantization on six SLMs from 0.5 to 3.8B, seven languages, and seven downstream tasks. |
Enjoy the Salience: Towards Better Transformer-based Faithful Explanations with Word Salience (2021.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained transformer-based language models have demonstrated state-of-the-art predictive performance when adapted into a range of language understanding tasks. |
| Approach: | They propose to use salient information extracted a priori from training data to complement the task-specific information learned by the model during fine-tuning on a downstream task. |
| Outcome: | The proposed model can provide more faithful explanations across four different feature attribution methods compared to vanilla BERT. |
Recognizing Sentence-level Logical Document Structures with the Help of Context-free Grammars (2020.lrec-1)
Copied to clipboard
| Challenge: | Current sentence boundary detectors split documents into sequentially ordered sentences without their dependencies. |
| Approach: | They propose a tool that segments sentences into tree structures to detect recursive structure . they retrain different constituency parsers to transform them into sentence segmenters . |
| Outcome: | The proposed tool can detect recursive structure in documents with a main clause and subordinate clauses . the proposed tool improves German dependency parsing by providing additional structural information. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
New Protocols and Negative Results for Textual Entailment Data Collection (2020.emnlp-main)
Copied to clipboard
| Challenge: | Natural language inference data has proven useful in benchmarking and as pretraining data for tasks requiring language understanding. |
| Approach: | They propose four alternative protocols to improve annotation quality and diversity . they use 8.5k-example training sets to compare different protocols . |
| Outcome: | The proposed protocols improve the ease of training and quality of the examples. |
TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition (2025.acl-long)
Copied to clipboard
Tianwei Lin, Jiang Liu, Wenqiao Zhang, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Jiannan Guo, Hao Jiang, Siliang Tang, Yueting Zhuang
| Challenge: | Existing methods for fine-tuning are resource-efficient, but performance often falls short . a new approach, TeamLoRA, integrates collaborative and competitive modules to improve performance. |
| Approach: | They propose to introduce task-specific LoRA as domain experts to improve learning efficiency . teamLoRA integrates collaborative and competition modules to improve model learning . |
| Outcome: | Experiments show that TeamLoRA improves performance in multi-task learning . teamLorea integrates collaborative and competitive modules to improve performance . |
How does the task complexity of masked pretraining objectives affect downstream performance? (2023.findings-acl)
Copied to clipboard
| Challenge: | Masked language modeling (MLM) is a widely used self-supervised pretraining objective. |
| Approach: | They propose to use a mask-based objective to predict a token that is replaced with a masked token given its context. |
| Outcome: | The proposed objectives show that they should have half the complexity needed to perform comparably to MLM. |
XMark: Reliable Multi-Bit Watermarking for LLM-Generated Texts (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for embedding binary messages into LLM-generated text suffer from key limitations, such as a poor trade-off between text quality and decoding accuracy. |
| Approach: | They propose a method for embedding binary messages into Large Language Model (LLM)-generated text that uses a limited number of tokens to decode and recover the encoded message. |
| Outcome: | The proposed method significantly outperforms existing methods in multiple downstream tasks and will be made publicly available upon acceptance. |
Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | A study of multilingual fine-tuning yields better performance on downstream NLP applications . low resource languages such as Oriya and Punjabi are found to be the largest beneficiaries of multi-lingual fine tuning. |
| Approach: | They propose to leverage the relatedness of languages that belong to the same family in NLP models by multilingual fine-tuning. |
| Outcome: | The proposed approach improves performance on downstream NLP tasks by 15% compared to monolingual fine-tuning. |
A Generative Pre-Trained Language Model for Channel Prediction in Wireless Communications Systems (2025.emnlp-main)
Copied to clipboard
Bo Lin, Huanming Zhang, Yuhua Jiang, Yucong Wang, Tengyu Zhang, Shaoqiang Yan, Hongyao Li, Yihong Liu, Feifei Gao
| Challenge: | Existing model-based channel prediction methods suffer from limited accuracy due to imperfect temporal modeling, while existing AI-based methods suffers from limited generalization due to inadequate training strategies. |
| Approach: | They propose a generative pre-trained language model for channel prediction based on channel correlation and train it based upon transformer decoder architecture. |
| Outcome: | The proposed model can learn various channel characteristics and perform impressive tasks across multiple dimensions. |
Bridging the Memorization-Utilization Gap: Near-Lossless Context Compression via Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in context compression have failed to effectively utilize compressed representations for downstream tasks. |
| Approach: | They propose a holistic training paradigm that uses outcome-based RL to enable implicit expansion. |
| Outcome: | The proposed model outperforms previous models on NIAH, LongBench and multi-hop reasoning. |
Cross-Modality Relevance for Reasoning on Language and Vision (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to learn and reason over language and vision data for downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR) |
| Approach: | They propose a cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under supervision of a target task. |
| Outcome: | The proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. |
Preventing Author Profiling through Zero-Shot Multilingual Back-Translation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Documents as short as a single sentence may reveal sensitive information about authors . style transfer is effective but a number of current methods cause a drop in down-stream utility . |
| Approach: | They propose a method to remove sensitive information from documents by multilingual back-translation using off-the-shelf translation models. |
| Outcome: | The proposed method lowers adversarial gender and race prediction by 22% while retaining 95% of original utility on downstream tasks. |
Understanding In-Context Learning via Supportive Pretraining Data (2023.acl-long)
Copied to clipboard
| Challenge: | In-context learning (ICL) is a form of learning that provides a handful of examples at inference time, but it is not well understood why it emerges as the model has never been specifically trained on such demonstrations. |
| Approach: | They adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL and compare it with random subsets of pretrain data. |
| Outcome: | The proposed method improves the model's ICL ability by 18% if it is continued on a small subset of pretraining data. |
Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent prompt-based approaches allow pretrained language models to achieve strong performances on few-shot finetuning by reformulating downstream task instances as a language modeling problem. |
| Approach: | They propose to reformulate downstream tasks as a language modeling problem and add a regularization that preserves pretraining weights to the model to mitigate the destructive tendency of few-shot finetuning. |
| Outcome: | The proposed model performs better on low data regimes than the standard model on few-shot finetuning. |
Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, including instruction following, mathematical problem solving, and coding generation. |
| Approach: | They propose a method that truncates both preferred and dispreferred responses to match the shorter one’s length. |
| Outcome: | The proposed approach improves over standard implementations and achieves 11.8 points in AlpacaEval 2 and overall improvements across downstream tasks. |
Once Upon a Time in Graph: Relative-Time Pretraining for Complex Temporal Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing work focuses on strengthening the knowledge-time association between text and time-stamps, but this is insufficient for downstream tasks. |
| Approach: | They propose a model that explicitly connects all temporally-scoped facts by modeling the time relations between any two sentences. |
| Outcome: | The proposed model outperforms baseline T5 on multiple temporal question answering datasets . it is especially good at modeling long-range complex temporal dependencies, the authors say . |
No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement (2025.coling-main)
Copied to clipboard
| Challenge: | Modular deep learning is the most effective way to lift the curse of multilinguality. |
| Approach: | They propose a method which enables training-free post-processing to address this limitation by adding learning to the language adapters and transitioning the framework from a multi-task to a multiple language setup. |
| Outcome: | The proposed method consistently improves baselines with significant gains, especially in the most challenging case of zero-shot application. |
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pretrained language models extend from millions to billions of parameters. |
| Approach: | They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process. |
| Outcome: | The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins. |
DirectQuote: A Dataset for Direct Quotation Extraction and Attribution in News Articles (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods to extract and attribute quotations from news data are difficult and require a lot of effort. |
| Approach: | They propose a corpus of 19,760 paragraphs and 10,279 direct quotations manually annotated from online news media. |
| Outcome: | The proposed corpus contains 19,760 paragraphs and 10,279 direct quotations manually annotated from online news media. |
Empowering Large Language Models for Textual Data Augmentation (2024.findings-acl)
Copied to clipboard
| Challenge: | True. True. False |
| Approach: | False slants are proposed to generate a large pool of augmentation instructions and select the most suitable task-informed instructions. |
| Outcome: | False omissions: the proposed approach consistently generates augmented data with better quality compared to non-LLM and LLM-based data augmentation methods. |
Pre-training Language Models for Comparative Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions. |
| Approach: | They propose a framework to pre-train language models for enhancing comparative reasoning abilities . they collect scalable data for text-based entity comparison . |
| Outcome: | The proposed framework significantly improves comparative reasoning abilities under low-resource conditions on downstream tasks. |
Masks Can be Learned as an Alternative to Experts (2025.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that sparse activation techniques can reduce inference performance without sacrificing performance. |
| Approach: | They propose to sparsify a pre-trained dense large language model into a mixture-of-experts architecture for faster inference. |
| Outcome: | The proposed approach is more efficient than one-shot sparsification techniques . it achieves 97% performance retention on downstream tasks with only 50% of parameters activated . |
EpiCoDe: Boosting Model Performance Beyond Training with Extrapolation and Contrastive Decoding (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to enhance performance of Large language models are limited due to the cost of training data and privacy concerns. |
| Approach: | They propose a method that enhances a finetuned model with its inferior version and adopts contrastive decoding to reduce predicted errors. |
| Outcome: | The proposed method outperforms existing methods in data-scarcity scenarios across three domains and shows that it is more robust and robust. |
M3Retrieve: Benchmarking Multimodal Retrieval for Medicine (2025.emnlp-main)
Copied to clipboard
| Challenge: | Strong retrieval models are increasingly important in knowledge-intensive domains. |
| Approach: | They propose a benchmark to evaluate multimodal retrieval models in medical settings . they examine 1.2 million text documents and 164K multimodal queries . |
| Outcome: | The proposed model spans 5 domains,16 medical fields, and 4 distinct tasks with over 1.2 Million text documents and 164K multimodal queries. |
Syntax-driven Approach for Semantic Role Labeling (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing studies focus on auto-generated syntactic knowledge to enhance semantic role labeling . experimental results show that map memories can enhance SRL . |
| Approach: | They propose to map memories to enhance semantic role labeling by encoding auto-generated syntactic knowledge from off-the-shelf toolkits. |
| Outcome: | The proposed model outperforms baselines and achieves state-of-the-art results on two English benchmark datasets. |
Entailment as Robust Self-Learner (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have found that entailment pretraining benefits weakly supervised fine-tuning. |
| Approach: | They propose a prompting strategy that formulates different NLU tasks as contextual entailment and propose an algorithm for better pseudo-labeling quality in self-training. |
| Outcome: | The proposed approach improves the zero-shot adaptation performance on downstream tasks. |
EconBERTa: Towards Robust Extraction of Named Entities in Economics (2023.findings-emnlp)
Copied to clipboard
Karim Lasri, Pedro Vitor Quinta de Castro, Mona Schirmer, Luis Eduardo San Martin, Linxi Wang, Tomáš Dulka, Haaya Naushan, John Pougué-Biyong, Arianna Legovini, Samuel Fraiberger
| Challenge: | Adapting general-purpose language models to specific domains has proven to be effective in tackling downstream tasks such as impact evaluation. |
| Approach: | They propose to use EconBERTa, a large language model pretrained on scientific publications in economics, and ECON-IE, based on an expert-annotated dataset of economics abstracts for Named Entity Recognition (NER). |
| Outcome: | The proposed model outperforms EconBERTa on the downstream NER task and ECON-IE on the economics abstracts. |
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained masked language models perform few-shot learning, but discriminative models like ELECTRA do not fit into the paradigm. |
| Approach: | They propose to use ELECTRA to train pre-trained models to score originality of target options without introducing new parameters. |
| Outcome: | The proposed model outperforms masked language models in a wide range of tasks without adding new parameters. |
Social Bias Evaluation for Large Language Models Requires Prompt Variations (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have tried to evaluate and mitigate social biases accurately using limited prompts. |
| Approach: | They investigate the sensitivity of Large Language Models when changing prompt variations . they found that LLM rankings fluctuate across prompts for both task performance and social bias . |
| Outcome: | The results show that LLM rankings fluctuate when changing prompt variations . |
CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained vision-language models perform well in cross-modal tasks, including referring expression comprehension. |
| Approach: | They propose a method that enables VL models to reason with implicit text . they propose to use a dataset to align the text with objects in the images . |
| Outcome: | The proposed method improves performance 37.94% on referring expression comprehension task. |
Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning (2025.findings-acl)
Copied to clipboard
| Challenge: | Experimental results show that PEFT can fine-tune language models without relying on perfectly labeled datasets. |
| Approach: | They propose a framework that decouples sample selection from model training by introducing clean and noisy LoRA. |
| Outcome: | The proposed framework decouples sample selection from model training. |
ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs (2024.findings-emnlp)
Copied to clipboard
Preetam Prabhu Srikar Dammu, Himanshu Naidu, Mouly Dewan, YoungMin Kim, Tanya Roosta, Aman Chadha, Chirag Shah
| Challenge: | Despite the fact that many fact-checking tools lack granularity and explainability, they lack the ability to be useful in various contexts. |
| Approach: | They propose a text validation framework that provides granular explanations for each claim and localizes the specific problematic content to reduce cognitive load. |
| Outcome: | The proposed framework provides granular explanations for each claim prediction and localizes and educates users on the specific content. |
FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual models have been released, but many of the world's languages are not covered. |
| Approach: | They propose a method that initializes the embedding matrix for a new tokenizer based on information in the source model's embeddable matrix. |
| Outcome: | The proposed method outperforms random initialization and previous work on language modeling and on a range of downstream tasks (NLI, QA, and NER). |
LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained models on document images with transformer-based backbones have led to significant performance gains in this field. |
| Approach: | They propose a multi-modal pre-training model that combines text, layout and image . they propose to use local 1D position instead of global 1D positions as layout input . |
| Outcome: | The proposed model can achieve state-of-the-art results on a wide variety of VrDU problems. |
POSIX: A Prompt Sensitivity Index For Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are sensitive to minor variations in prompts, such as spelling errors, alteration of wording or the prompt template. |
| Approach: | They propose a PrOmpt Sensitivity IndeX to measure prompt sensitivity . they use this to compare prompt sensitability of various open source LLMs . |
| Outcome: | The proposed method can measure and compare prompt sensitivity of open source LLMs. |
The Benefits of Label-Description Training for Zero-Shot Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data to classify among specific label sets in downstream tasks. |
| Approach: | They propose to use a small finetuning dataset to describe the labels for a task and to use it to further improve zero-shot accuracies. |
| Outcome: | The proposed model is more accurate than zero-shot by 17-19% absolute across topic and sentiment datasets and more robust to choices required for zero- shot classification. |
MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models can encode knowledge during pre-training on large text corpora, enabling downstream tasks like question answering (QA). |
| Approach: | They construct a dataset derived from systematic reviews to examine their ability to encode medical knowledge and their recall. |
| Outcome: | The proposed model performs well on the biomedical QA dataset. |
WildReward: Learning Reward Models from In-the-Wild Human Interactions (2026.acl-long)
Copied to clipboard
| Challenge: | Prior work focused on collecting preference pairs, requiring substantial annotation efforts. |
| Approach: | They propose a pipeline to extract reliable human feedback from in-the-wild interactions . they propose to use WildChat as an interaction source to train the model . |
| Outcome: | The proposed model achieves comparable or even superior performance compared to conventional models with improved calibration and cross-sample consistency. |
NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing backdoor attacks against prompt-based learning involve injecting back doors into embedding layers or word embedders. |
| Approach: | They propose a backdoor attack against prompt-based learning that injects backdoors into embedding layers or word embeddable vectors. |
| Outcome: | The proposed backdoor attack outperforms two state-of-the-art models on six NLP tasks and three prompting strategies. |
Familiarity-Aware Evidence Compression for Retrieval-Augmented Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) improves large language models by incorporating non-parametric knowledge through evidence retrieved from external sources. |
| Approach: | They propose a training-free evidence compression technique that makes retrieved evidence more familiar to the target model while seamlessly integrating parametric knowledge from the model. |
| Outcome: | The proposed technique outperforms the most recent evidence compression baselines across open-domain QA datasets while achieving high compression rates. |
Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)
Copied to clipboard
Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency
| Challenge: | Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes. |
| Approach: | They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. |
| Outcome: | The proposed masking strategy outperforms baselines on language-only and multimodal video tasks. |
From Speculation Detection to Trustworthy Relational Tuples in Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on speculation detection are defined at sentence level, but not all factual tuples extracted from a sentence are speculative. |
| Approach: | They propose to study speculations in OIE tuples and determine whether a tample is speculative. |
| Outcome: | The proposed model is based on the LSOIE dataset and provides labels for speculative tuples. |
Each graph is a new language: Graph Learning with LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Natural language is used to describe graphs, but graph descriptions become verbose and only relying on attribute embeddings limits LLM’s ability to capture adequate graph structural information. |
| Approach: | They propose a graph-defined language for large language model that translates the graph into a corpus instead of graph descriptions and pre-trains LLMs on this corpus to adequately understand the graph. |
| Outcome: | Experiments on five datasets show that the proposed framework outperforms description-based and embedding-based baselines by efficiently modeling different orders of neighbors. |
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for chart-based data analysis neglect explicit modeling of chart structures. |
| Approach: | They propose a pretrained model for chart comprehension and reasoning that encodes relevant text, data, and visual elements of charts and uses a chart-grounded text decoder for text generation. |
| Outcome: | The proposed model outperforms existing methods that lack explicit modeling of chart structures and lacks explicit modeling. |
Task-Aware Self-Supervised Framework for Dialogue Discourse Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing discourse parsing approaches are constrained by predefined relation types, which can impede the adaptability of the parser for downstream tasks. |
| Approach: | They propose to introduce a task-aware paradigm to improve the versatility of the parser. |
| Outcome: | Empirical studies on dialogue discourse parsing datasets and a downstream task demonstrate the proposed framework. |
Multi-label Sequential Sentence Classification via Large Language Model (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to sequential sentence classification are constrained by model size, sequence length, and single-label setting. |
| Approach: | They propose a large language model-based framework for both single- and multi-label SSC tasks that generate SSC labels through designed prompts. |
| Outcome: | The proposed framework enhances task understanding by incorporating demonstrations and a query to describe the prediction target. |
Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectives (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown increasing power on NLP tasks. however, tuning these models for downstream tasks usually requires exorbitant costs. |
| Approach: | They propose a black-box tuning technique that optimizes task-specific prompts without accessing gradients and hidden representations. |
| Outcome: | The proposed method improves performance under few-shot learning scenarios. |
MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) have demonstrated impressive molecule understanding ability on 1D text-related tasks, but lack 2D graph perception, a critical ability of human professionals in comprehending molecules’ topological structures. |
| Approach: | They propose to combine a cross-modal projector and a uni-modal adapter to enable an LM to understand both text- and graph-based molecular contents via a Q-Former. |
| Outcome: | The proposed model outperforms the baselines on tasks such as molecule captioning, IUPAC name prediction, and molecule-text retrieval. |
TransferCVLM: Transferring Cross-Modal Knowledge for Vision-Language Modeling (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large vision-language multimodal models pre-trained with huge amount of image-text pairs show remarkable performances in downstream tasks. |
| Approach: | They propose a method of efficient knowledge transfer that integrates pre-trained uni-modal models into a combined vision-language model without pre-training . they propose to fine-tune the model and transfer multimodal knowledge from a teacher vision-linguistic model to the CVLM for each task application. |
| Outcome: | The proposed method outperforms existing vision-language models in downstream tasks. |
Learning Subjective Label Distributions via Sociocultural Descriptors (2025.emnlp-main)
Copied to clipboard
| Challenge: | Conventional approaches aggregate annotator judgements (labels), ignoring minority perspectives, and overlooking the influence of sociocultural context behind such annotations. |
| Approach: | They propose a framework where subjectivity in binary labels is modeled as an empirical distribution accounting for the variation in annotators through human values extracted from sociocultural descriptors using a language model. |
| Outcome: | The proposed model yields well-calibrated toxicity distribution predictions across binary toxicity labels, which are further used for majority label prediction across cultural subgroups. |
MultiLeg: Dataset for Text Sanitisation in Less-resourced Languages (2024.lrec-main)
Copied to clipboard
| Challenge: | Text sanitization is the task of detecting and removing personal information from the text. |
| Approach: | They propose a dataset for multilingual named entities that can be used for text sanitization. |
| Outcome: | The proposed dataset is available in 8 languages and contains 3082 parallel text segments for each language. |
How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads (2025.emnlp-main)
Copied to clipboard
| Challenge: | Despite advances in Large Vision Language Models, a gap remains in their interpretability and performance. |
| Approach: | They identify the Optical Character Recognition Head (OCR Head) heads that are more efficient at recognizing text from images. |
| Outcome: | The Optical Character Recognition Head (OCR Head) is identified as the most efficient head for recognizing text from images. |
ExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown great power in solving various tasks but fail in many specific tasks. |
| Approach: | They propose a framework to help black-box LLMs better adapt to unfamiliar tasks by reflecting and noting experiences from training data and retrieving them from external memory during testing. |
| Outcome: | The proposed framework improves the performance of black-box Large Language Models on multiple tasks and demonstrates that it is a good choice for the future. |
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models are trained on vast unlabeled data, rich in world knowledge. |
| Approach: | They propose a categorization scheme for factual probing methods based on how inputs, outputs and probed PLMs are adapted . they synthesize insights about knowledge retention and prompt optimization in PLM models and analyze obstacles to adopting them as knowledge bases . |
| Outcome: | The proposed method synthesizes insights about knowledge retention and prompt optimization in PLMs, analyzes obstacles to adopting them as knowledge bases and outline directions for future work. |
EcoLoRA: Communication-Efficient Federated Fine-Tuning of Large Language Models (2025.emnlp-main)
Copied to clipboard
Han Liu, Ruoyao Wen, Srijith Nair, Jia Liu, Wenjing Lou, Chongjie Zhang, William Yeoh, Yevgeniy Vorobeychik, Ning Zhang
| Challenge: | Recurrent exchange of model updates in FL can result in prohibitively high communication costs, hindering the distributed learning process. |
| Approach: | They propose a federated fine-tuning framework that uses a round-robin segment sharing scheme to reduce network bandwidth and adaptive sparsification methods tailored to LoRA’s training dynamics. |
| Outcome: | The proposed framework reduces communication overhead without compromising performance on question-answering and value-alignment tasks. |
Generative Annotation for ASR Named Entity Correction (2025.emnlp-main)
Copied to clipboard
Yuanchang Luo, Daimeng Wei, Shaojun Li, Hengchao Shang, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Xiaoyu Chen, Zhiqiang Rao, Jinlong Yang, Hao Yang
| Challenge: | Existing named entity correction models fail to transcribe domain-speciffcnamed entities when theforms of the wrongly-transcribed words and the ground-truth entity are signiffcantly different. |
| Approach: | They propose a method that utilizes speech sound features to retrieve candidate entities . it uses speech sound feature to annotate entityerrors in ASR transcripts . |
| Outcome: | The proposed method can bring signiffcant improvement to entity accuracy. |
Efficient Unseen Language Adaptation for Multilingual Pre-Trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Multilingual pre-trained language models (mPLMs) have demonstrated notable effectiveness in zero-shot cross-lingual transfer tasks. |
| Approach: | They propose a method that uses soft-prompt tuning to tune for language adaptation . prompt tuning outperforms continuously trained baselines on two benchmarks . |
| Outcome: | The proposed approach outperforms baselines on two text classification benchmarks while utilizing 0.28% of tuned parameters. |
DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have significantly enhanced their knowledge and generative capabilities, leading to a surge of interest in leveraging LLMs for high-quality data synthesis. |
| Approach: | They propose a controllable data synthesis framework based on variational autoencoder which leverages diffusion models to reserve more information of original distribution and format structure in the learned latent distribution. |
| Outcome: | The proposed framework generates high-quality data with performance exceeding that of real data by 2%–7% on seven real-world datasets. |
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. |
| Approach: | They propose a pruning-based approach to balance CF and downstream task performance by integrating the ratio of the task vector to pre-trained model parameters into the pruning criteria. |
| Outcome: | The proposed pruning-based approach limits CF to just 0.25% while maintaining 99.67% accuracy on downstream tasks. |
The Role of Outgoing Connection Heterogeneity in Feedforward Layers of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Using a new fine-tuning loss, we show that inner neurons with diverse outgoing connections are more critical to model performance than those with uniform connections. |
| Approach: | They propose a new loss that reduces the outgoing connection entropy in feedforward layers and elucidates the role of outgoing connections in large language models. |
| Outcome: | The proposed method is significantly more effective than removing neurons randomly or based on their magnitude. |
Don’t Half-listen: Capturing Key-part Information in Continual Instruction Tuning (2025.acl-long)
Copied to clipboard
Yongquan He, Wenyuan Zhang, Xuancheng Huang, Peng Zhang, Lingxun Meng, Xiang Zhou, Ke Zeng, Xunliang Cai
| Challenge: | Existing methods to improve instruction tuning for large language models may cause catastrophic forgetting (CF) CF is a problem where previously learned abilities are degraded . |
| Approach: | They propose a continual instruction tuning method that uses key-part information gain to replay data and refine training objective. |
| Outcome: | The proposed method achieves superior performance on both seen and held-out tasks. |
Capability Salience Vector: Fine-grained Alignment of Loss and Capabilities for Downstream Task Scaling Law (2025.acl-long)
Copied to clipboard
Qiming Ge, Shuhao Xing, Songyang Gao, Yunhua Zhou, Yicheng Zou, Songyang Zhang, Zhi Chen, Hang Yan, Qi Zhang, Qipeng Guo, Kai Chen
| Challenge: | Large language models have demonstrated impressive performance across a wide range of tasks, but this achievement comes with the trade-off of significant computational demands. |
| Approach: | They propose a scaling law that decomposes the overall validation loss and assigns different importance weights to tokens to assess a specific meta-capability. |
| Outcome: | The proposed model can predict the loss trending of models across different levels of computation without a gap between validation loss and model's downstream capabilities. |
Task Oriented In-Domain Data Augmentation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models suffer from two major issues: in-domain data are scarce compared with general domain-agnostic data. |
| Approach: | They propose a task-oriented in-domain data augmentation framework that uses in- domain data selection and task-orientated synthetic passage generation to adapt LLMs to two domains: advertisement and math. |
| Outcome: | The proposed framework improves LLM performance by 8% in the advertisement domain and 7.5% in the math domain. |
Language Fusion for Parameter-Efficient Cross-lingual Transfer (2025.acl-long)
Copied to clipboard
| Challenge: | Limited availability of multilingual text corpora for pretraining results in poor performance on downstream tasks due to undertrained representation spaces for languages other than English. |
| Approach: | They propose a method that integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations to enhance representation quality and transfer performance for languages other than English. |
| Outcome: | The proposed method improves representation quality and performance for languages other than English while maintaining parameter efficiency. |
HomoGraphAdapter: A Homogeneous Graph Neural Network as an Effective Adapter for Vision-Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing adaptation methods overlook structural knowledge between text and image modalities or create overly complex graphs containing redundant information for alignment. |
| Approach: | They propose a method to adapt visual models to downstream tasks using text and image modalities. |
| Outcome: | The proposed method improves classification accuracy by 1.51% for 1-shot and 0.74% for 16-shot on 11 datasets. |
Difference in Task Performance on Sparse Speech Representations (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for learning speech representations that are useful for a variety of downstream tasks have been extensively investigated in different domains. |
| Approach: | They propose to train Autoencoders with varying sparsity levels using three SSL features and evaluate them on six tasks of SUPERB: speech enhancement, speaker identification, speech Emotion Recognition, phone recognition, automatic speech recognition and slot filling. |
| Outcome: | The proposed model can be used to learn speech representations that are useful for a variety of downstream tasks. |
GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment (2026.acl-long)
Copied to clipboard
Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Lin Hai, Yiming Zhao, Hai-Tao Zheng, Hong-Gee Kim
| Challenge: | Large Language Models (LLMs) have achieved remarkable performance across NLP tasks . however, in long-context scenarios, they face high computational cost and information redundancy. |
| Approach: | They propose an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. |
| Outcome: | Experiments show that GMSA outperforms baselines on multiple long-context question answering and summarization benchmarks while maintaining low end-to-end latency. |
Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptions (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies show that direct generation of diagram descriptions is costly and biased against blind and low-vision (BLV) users. |
| Approach: | They ask sighted individuals to assess diagram descriptions generated by vision-language models . they use latent supervision to guide the models with latent inference . |
| Outcome: | The results show that visual descriptions generated by vision-language models are effective and useful to educators who are themselves BLV and teach visually impaired learners. |
Targeted Syntactic Evaluation on the Chomsky Hierarchy (2024.lrec-main)
Copied to clipboard
| Challenge: | a novel evaluation paradigm for targeted syntactic evaluations is proposed . we create formal languages that abstract four syntaktic phenomena in natural languages . |
| Approach: | They propose a new evaluation paradigm for Targeted Syntactic Evaluations . they create formal languages that abstract syntactical phenomena in natural languages . |
| Outcome: | The proposed evaluation paradigm evaluates language models on language modeling tasks . it shows that they can capture the structural patterns of the (Adj)n NP type formal language . |
Sticking to the Mean: Detecting Sticky Tokens in Text Embedding Models (2025.acl-long)
Copied to clipboard
| Challenge: | Sticky tokens, when repeatedly inserted into sentences, pull sentence similarity toward a certain value, disrupting the normal distribution of embedding distances and degrading downstream performance. |
| Approach: | They propose a method to detect “sticky tokens” by sentence and token filtering and apply it to 40 checkpoints across 14 model families. |
| Outcome: | The proposed method detects 868 sticky tokens across 14 models and shows that their presence does not correlate with model size or vocabulary size. |
Length-Induced Embedding Collapse in PLM-based Models (2025.acl-long)
Copied to clipboard
| Challenge: | In text embeddings from PLMs are essential for many NLP applications, but performance degrades on longer texts. |
| Approach: | They propose a method which mitigates the phenomenon of Length Collapse . they propose TempScale to ensure more consistent embeddings across different text lengths . |
| Outcome: | The proposed method improves performance on MTEB and LongEmbed by 0.94% on short and 1.10% on long texts. |
Robust Utility-Preserving Text Anonymization Based on Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Existing techniques face challenges of re-identification ability of large language models . anonymizing text that contains sensitive information is crucial for a wide range of applications . |
| Approach: | They propose a framework that integrates three key LLM components to perform anonymization. |
| Outcome: | The proposed model outperforms baselines while maintaining greater data utility in downstream tasks. |
WikiSplit++: Easy Data Refinement for Split and Rephrase (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing text simplification methods rely on encoder-decoder models to achieve this task. |
| Approach: | They propose a text-to-text generation approach that applies encoder-decoder models to a large-scale dataset to improve Split and Rephrase. |
| Outcome: | The proposed approach improves Split and Rephrase readability and performance on large datasets, but still suffers from hallucinations and under-splitting. |
All Roads Lead to Rome: Graph-Based Confidence Estimation for Large Language Model Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for confidence estimation are primarily designed for factual QA tasks and fail to generalize to reasoning tasks. |
| Approach: | They propose a set of training-free, graph-based confidence estimation methods tailored to reasoning tasks that exploit graph properties such as centrality, path convergence, and path weighting. |
| Outcome: | The proposed methods improve confidence estimation and performance on two downstream tasks. |
GraphSynth: Resolving the Diversity-Reliability Trade-off with Probabilistic Factor Graphs (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are a scaleable solution for the generation of synthetic data . however, the utility of such data is capped by a critical tension between diversity and factual reliability. |
| Approach: | They propose a framework which leverages a probabilistic factor graph modeling the universe of attributes. |
| Outcome: | The proposed framework outperforms state-of-the-art models with a high structural integrity and a boost in performance on downstream tasks. |
Uncovering Currency Bias and Syntax Gap in Text Embedding Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Text-embedding models often inherit societal biases, yet the influence of socio-economic markers remains unexplored. |
| Approach: | They propose to identify Currency Bias as a systemic representational limitation in financial AI . they analyze currency embeddings to identify currency identifiers and associative sensitivity . |
| Outcome: | The proposed model lacks associative sensitivity to economic hierarchies, the authors show . they show that current embedding practices pose significant risks for the fairness and reliability of financial NLP applications. |
NSF-SciFy: Mining the NSF Awards Database for Scientific Claims (2026.acl-long)
Copied to clipboard
| Challenge: | NSF-SciFy contains 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines. |
| Approach: | They propose to use a dataset to extract scientific claims from National Science Foundation award abstracts and to use it to refine language models. |
| Outcome: | The proposed method improves non-technical abstract generation, claim extraction, and investigation proposal extraction tasks while maintaining high precision and lower recall. |